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ABSTRACT 


\ 

— The  problem  of  optimization  of  stochastic  dynamic  systems  with  random 
coefficients  is  discussed.  Systems  with  both  Wiener  processes  and  uncertain 
random-process  disturbances  are  dealt  with^  i-n^thi s  report  and  these  include 
certain  bilinear  stochastic  systems.  It  is  the  purpose  ef— fch4s  to 

study  the  optimal  control  and,  to  some  extent,  state  estimation  of  such  bi¬ 
linear  stochastic  systems.  By  means  of  the  stochastic  Bellman  equation,  the 
optimal  control  of  stochastic  dynamic  models  with  observable  and  unobservable 
coefficients  is  derived. 

The  stochastic-system  model  considered  is  the  observable  system  with 
random  coefficients  that  are  a  function  of  the  solution  of  a  certain  unob¬ 
servable  Markov  process  with  information  data.  Under  the  assumptions  that  the 
solution  of  the  stochastic  differential  equation  for  the  dynamic  model  in¬ 
volved  in  the  problem  formulation  results  in  an  admissible  control  and  that 
the  measurable  information  of  all  random  parameters  depend  on  the  conditional- 
mean  estimate  to  the  unobservable  stochastic  process,  the  optimal  control  is  a 
linear  function  of  the  observable  states  and  a  nonlinear  function  of  random 
parameters. 

\ 

The  theory  is  then  applied  to  an  optimal -control  design  of  an  aircraft 
landing  with  a  bad  weather  situation,  to  the  control  problem  of  longitudinal 
motion  of  an  aircraft  in  wind  gust,  and  to  nonlinear  filtering  and  tracking  of 
the  maneuvering  target.  The  flare  path  compared  with  the  desirable  exponen¬ 
tial-linear  path  provides  a  safe  and  comfortable  landing  for  the  optimal- 
control  policy.  Using  the  decoupled  feedback  law  of  longitudinal  aircraft 
motion,  it  is  shown  that  optimal -control  policies  for  the  elevator  control 
angle  and  the  aileron  control  angle  Is  synthesized  with  the  attack  angle,  the 
orientation  rate  of  the  aircraft  and  an  unknown  random  parameter.  In  the 


final  example,  a  maneuvering  target's  state  is  estimated.  Here,  a  bilinear 
stochastic  model  is  assumed  such  that  discrete  velocity  changes  are  at  random 


times. 
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OPTIMIZATION  OF  STOCHASTIC  DYNAMIC  SYSTEM  WITH 
RANDOM  COEFFICIENTS 

1.  INTRODUCTION 

Most  of  the  results  in  stochastic  dynamic  control  and  filter¬ 
ing  theory  were  obtained  with  the  assumptions  that  the  process 
under  some  special  considerations  satisfy  stochastic  differential 
equations.  Unfortunately,  very  little  is  known  about  optimal  con¬ 
trol  theory  for  processes  governed  by  general  stochastic  differ¬ 
ential  equations.  In  this  area,  most  of  the  contributions  are 
made  by  Fleming  [1,2, 3, 4],  Kushner  [5,6],  Wonham  [7,8,9],  Balakrish 
nan  [10,11],  Benes  [12,13],  Rishel  [14,15],  Bismut  [16,17,18],  Davis 
[19,20,21,22,23],  Eliott  [24,25],  Haussman  [26,27],  Variya  [28], 
Kolodziej  [29],  and  Mohler  and  Kolodziej  [30,31]. 

The  status  of  continuous-time  stochastic  theory  was  summarized 
in  Fleming's  1969  survey  paper  [3]  and  also  introduced  in  his  book 
[4]  concerning  the  control  of  completely  observable  diffusion  pro¬ 
cesses.  In  regard  to  problems  with  partial  observation,  possibly 
the  most  significant  result  was  from  Wonham' s  formulation  of  the 
separation  principle  [7]  using  the  stochastic  version  of  Bellman's 
dynamic  programming  which  Wonham  proved  by  reformulating  the  problem 
as  one  of  the  complete  observations  with  the  state  being  the  con¬ 
ditional  mean  estimate  produced  by  the  Kalman-Bucy  filter. 

The  dynamic-programming  method  is  a  useful  approach  in  sto¬ 
chastic  control.  However,  these  conditions  under  dynamic  program¬ 
ming  are  so  much  weaker  than  those  required  in  deterministic  control 
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The  dynamic-programming  approach,  while  successful  in  many  appli¬ 
cations,  suffers  from  many  limitations.  An  immediate  one  is  that 
the  controls  have  to  be  smooth  functions  of  the  state  in  order  that 
the  stochastic  differential  equation  has  a  solution  in  the  Ito 
sense. 

In  problems  of  engineering  design,  it  is  often  necessary  to 
choose  component  values  from  an  admissible  set  in  addition  to 
choosing  control  from  an  admissible  class,  such  that  the  final 
product  has  the  best  performance  in  some  appropriate  sense.  A 
typical  example  is  the  stochastic  dynamic  system  with  random  co¬ 
efficients  which  has  attracted  the  attention  of  numerous  authors 
[9,15,16,17,18,29,30,31,32,33,34,35,36,37].  One  of  the  most  formal 
approaches  to  stochastic  control  of  such  systems  was  presented  by 
Bismut  on  convex  analysis.  Bismut  solved  a  very  general 
class  of  linear,  quadratic,  finite-dimensional,  stochastic-control 
problems  with  random  coefficients  in  which  both  state  and  control 
dependent  noises  are  admitted,  and  he  also  discussed  the  necessary 
condition  of  stochastic  optimality.  Bensoussan  and  Voit  [38]  have 
considered  optimal  control  problems  for  the  stochastic  evolution 
equation  with  deterministic  operator-valued  coefficients  and  have 
developed  a  separation  principle  for  the  problem.  Recently,  Ahmed 
[35]  has  solved  the  optimal-control  porblem  of  the  stochastic  evolu¬ 
tion  equation  and  has  considered  a  random  operator  Riccati  equation 
and  backward  stochastic  evolution  equation.  For  the  stochastic 
discrete-time  system,  the  linear-quadratic  optimal  control  has  been 
investigated  by  Athans,  et  al.,  [36]  and  Ku,  et  al.,  [37]. 
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Kolodziej  [29],  Mohler  and  Kolodziej  [30,31]  presented  an 
alternate  approach  to  the  stochastic  control  of  a  class  of  linear 
stochastic  systems  with  random  coefficients.  Kolodziej  solved  the 
problem  of  optimal  control  using  dynamic  programming .  He  also  dis¬ 
cussed  the  separation  of  filtering  and  control  which  was  proven 
by  reformulating  the  problem  as  one  of  complete  observations  and 
the  control  problem  as  an  optimal  regulator,  which  is  a  linear  func¬ 
tion  of  the  unobservable  part  of  the  process  and  a  nonlinear  function 
of  the  observable  parts.  Sufficient  conditions  for  optimal  control 
were  expressed  through  the  existence  of  a  bounded  solution  to  a  cer¬ 
tain  Cauchy  problem  for  a  parabolic  type  of  partial  differential 
equation.  He  assumed  that  the  random  process  is  conditionally 
Gaussian,  i.e. ,  the  conditional  distributions  on  the  given  random 
process  are  Gaussian  (P-a.s.). 

This  dissertation  presents  the  problem  of  the  optimal  control 
of  stochastic  differential  equations  with  random  coefficients.  Sys¬ 
tems  with  both  Wiener  processes  and  uncertain  random  process  dis¬ 
turbances  are  dealt  with  in  this  study,  and  these  include  certain 

r 

bilinear  stochastic  systems  [31,39,40,41].  It  is  the  objective 
of  this  thesis  to  study  the  optimal  control  and,  to  some  extent, 
state  estimation  of  such  bilinear  stochastic  systems. 

In  general,  it  is  difficult  for  a  natural  stochastic  system 
to  keep  the  conditional-Gaussian  distribution  of  some  states.  Hence, 
Chapter  2  of  this  thesis  considers  observable  stochastic-control 
systems  with  observable  finite-dimensional  random  coefficients. 

Here,  the  stochastic  Bellman  equation  to  diffusion  processes  is 


I  I  iiirtilfrtrtlillM^Seih  i  nee— »■ 


used  to  find  the  optimal-control  law  of  the  given  system.  This 
study  results  in  a  similar  form  of  the  optimal-control  law  [31) . 

The  system  model  considered  in  Chapter  3  is  the  observable 
stochastic  system  with  random  coefficients  that  are  a  function  of 
the  solution  of  a  certain  unobservable  Markov  process  with  the  in¬ 
formation  data.  The  results  are  shown  for  the  suboptimal  control 
of  the  above  special  stochastic  model.  All  random  parameters  of 
the  approximate  models  are  replaced  by  the  other  random  parameters 
that  depend  on  the  conditional-mean  estimate  of  the  Kalman-Bucy  fil¬ 
ter  to  the  unobservable  stochastic  process.  It  is  assumed  that 
the  solution  of  the  stochastic  differential  equations  in  all  cases 
results  in  an  admissible  control.  The  results  for  the  problem  are 
similar  to  optimal  control  with  a  Ricatti-like  equation,  but  the 
methodology  and  the  system  models  are  somewhat  different. 

The  application  of  the  theoretical  results  of  this  thesis  to 
physical  systems  is  presented.  The  pseudo-Wiener  process  is  gen¬ 
erated  by  a  Bernoulli  time  series  using  pseudo-uniform  random  num¬ 
bers  and  pseudo-Gaussian  random  numbers  N(0,1).  The  numerical  solu¬ 
tion  to  a  nonlinear  partial  differential  equation  is  found  by  the 
method  of  lines  [421 ,  and  the  simulation  studies  result  in  the  opti¬ 
mal  control  to  simple  stochastic  systems.  An  aircraft  landing  model 
in  a  gusty  wind  [34,43,44,45]  is  studied  here.  The  uncertain 
quantities  of  the  landing  model  may  be  presented  by  the  wind  gust 
of  the  Dryden  model  [46] .  The  simulation  results  suggest  the  al¬ 
titude  of  the  aircraft  during  the  landing  period.  The  flare  path 
compared  with  the  desirable  exponential-linear  path  provides  a  safe 
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and  comfortable  landing  for  the  optimal  control  policy.  The  longi¬ 
tudinal  motion  of  aircraft  in  a  gusty  wind  given  in  Chapter  4  can 
be  obtained  by  theoretical  results  of  Chapter  3. 

The  decoupled  feedback  law  of  longitudinal  motion  of  aircraft 
which  included  unknown  quantities  and  is  also  subject  to  uncertain 
noise  is  discussed.  Using  an  approximate  stochastic  model  of  longi¬ 
tudinal  aircraft  motion  for  the  worst  situation,  it  is  shown  that 
optimal  control  policies  for  the  elevator  control  angle  and  the 
aileron  control  angle  is  synthesized  with  the  attack  angle,  the 
orientation  rate  of  aircraft,  and  an  unknown  random  parameter.  The 
simulation  results  are  compared  with  the  assumptions  of  observable 
and  unobservable  cases,  respectively. 

In  Chapter  5,  anti-submarine  target-motion  analysis  on 
nonlinear  filtering  and  tracking  is  presented.  A  common  method 
in  the  target-tracking  problem  [47,48,49,50,51,52]  is  to  model  the 
target  dynamics  in  a  rectangular-coordinate  system  which  results  in 
a  linear  set  of  state  equations  because  of  the  assumption  that  drag 
forces  are  linear  relationships  in  states  or  are  neglected.  However, 
the  drag  forces  are  proportional  to  velocities  squared  in  each  tar¬ 
get  motion's  directions  [53,54].  Hence,  a  more  appropriate  mathe¬ 
matical  model  is  derived  for  the  consideration  of  nonlinear  target 
dynamics  underwater.  With  this  model,  this  chapter  compares  perform¬ 
ances  between  an  extended  Kalman  filter  and  a  truncated  second-order 
nonlinear  filter  as  applied  to  bearing-only-target  tracking.  In  the 
existence  of  a  maneuvering  of  target,  many  authors  [47,48,49,50,55, 
56,57,58,59,60,61,62,63]  have  proposed  compensators  for  the  Kalman 
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filter  for  adaptation  to  maneuvering  situation.  The  target  motion 
equation  for  estimates  of  the  vehicle  maneuvering  performance  may 
be  represented  by  a  bilinear  stochastic  system  which  has  jumps  so 
that  between  jumps  it  remains  in  specific  states  at  random  time  T. 
The  stochastic  process  involved  in  the  maneuver  is  discussed.  The 
last  section  shows  the  filtering  and  control  problem  as  the  adaptive 
control  is  a  function  of  random  time  T  and  the  new  extended  states 
by  solving  certain  Riccati  equations. 


... 


Notation 


(ft,F,P)  Complete  probability  space 

x(t)  Value  at  a  particular  elementary  event  ii  £  fi 
{x(t)}  Stochastic  vector  process 
E(xfc)  Expectation  of  xt 

E(xt|yt)  Conditional  expectation  with  respect  to  a  given  observation 
measure  yt 

Cov(x  x  )  Covariance  of  x  and  x^ 
st  st 

Var  (x  )  Variance  of  x 
s  s 

xfcn  Sequence  n«l ,2, . . . , 
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II A II  The  Euclidean  norm  is  defined  as  A  being  a  vector  or 

a  matrix  as  follows 

II A II 2  *  tr(AA*) ; 

F  Sub-cr-field  of  F 

/  a-algebra  {yg,  0  <_  s  <_t}  generated  by  observable  sto¬ 
chastic  process  {y  ,  t  £  [0,T] } 

Zfc  a-algebra  {x  and  y  ,  0  <_  s  <_  t}  generated  by  the  observ¬ 

able  stochastic  process  i and  y^,  t  c  [0,T]  } 

B (t, • )  Measurable  nonanticipative  functional  parameter 


inf 

infimum 

dx 

X 

Denotes 

dt 
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2.  OPTIMAL  CONTROL  OF  A  STOCHASTIC  SYSTEM  WITH 
RANDOM  COEFFICIENTS 

Let  (Q,F,P)  be  a  complete  probability  space;  F^,  an  nondecreas¬ 
ing  family  of  sub-a-algebras  of  F,  t  e  [0,T]  ;  (wt),  an  F^adapted 
Wiener  process  of  dimension  Z.  Let  xt  and  z ,  t  £  (0,TJ,  (dimen¬ 
sions  n  and  m,  respectively) ,  be  observable  continuous  processes 
satisfying: 

d  x.  ■  A(t,  z  )x,_  dt  +  BttjZ.)^  dt  +  G(t,z  )dw  ,  (2-1) 

d  zt  *  C (t,zfc)dt  +  D(t,zt)dw  ,  (2-2) 

x(0)  ■  x  ,  z(0)  *  z  . 

0  0 

The  random  variables  x  and  z  are  assumed  to  be  independent  of 

0  0 

w  »  t  e  [0,t] .  Let  F  be  the  a-algebra  in  the  finite  space  C  of  the 

t  L  X 

continuous  finite  functions,  9  =  {9  ,  s  <  t},  t  <  T.  Each  of  the 

s  —  — 

functionals  A(t,0),  B(t,0),  G(t,0),  C(t,0),  D(t,0)  is  assumed  to  be 

measureable  with  respect  to  Ffc  and  have  the  dimensions  n  x  n,  n  x  p, 

n  x  Z,  mxl,  m  x  Z,  respectively. 

The  control  ufc  is  assumed  to  be  of  a  Markov  type,  i.e.,  ufc  = 
u(*  ,z  ,t).  Then,  the  problem  is  to  find  a  control  in  an  admissible 
control  set  U  that  minimizes  the  average  cost  functional 

J(u)  *  E[/*  L(t,xt,ufc)dt] .  (2-3) 

2.1  Optimal  Control  with 
Complete  Measurement  Information 

If  the  stochastic  differential  equation  (2-1)  and  (2-2)  satisfy 
the  assumptions  of  the  theorem  2.1,  then  the  unique  solution  of  (2-1) 


and  (2-2)  exists  [47,48]. 


Theorem  2.1  Let  A(s,n),  B(s,n),  G(s,n),  C(s,n),  D(s,n), 
s  e  [0,T]  be  F^measurable  functionals,  C(s,n),  D(s,n)  satisfy 

the  Lipschitz  condition,  and  |A(t,n)  <k  <«,  | B ( t , n )  |  <^k  <  «, 

T  2  (xol  1  2 

f  llG(t,h)ll  <  k  <  °°.  Then,  if  is  a  F  -measurable  random 

0  -  3  lz  J  0 

*  *  U 

vector,  E[xq  XQ1  +  £[zQ  zgl  c the  stochastic  differential  equa¬ 
tions  (2-1)  and  (2-2)  have  the  unique  solutions,  i.e. ,  x^,  z^  are 
F^- adapted  and  x^,  z^  are  given  by 

(2-4) 

xt  =  x0  +  f0  Xs  dS  +  f0  B(s'2s)usds  +  ftQ  G(s'Zs)dV 

z.  =  z_  +  C (s ,z  )ds  +  /*  D(s,z  )dw  (2-5) 

too  s  0  s  s 

The  above  integrals  are  defined  in  the  Ito  sense.  Here  k^,  k2»  k3 
are  positive  constants. 

Proof  The  proof  is  omitted.  The  reader  is  referred  for 
details  to  Liptser  and  Shiryayev  [47,48].  > 

Now  consider  the  problem  of  optimal  control  of  x  for  t  e  [0, T] 
based  on  the  complete  observations  of  xfc  and  zfc.  The  observable 
states  xfc  and  zfc  are  the  solution  of  equations  (2-1)  and  (2-2) , 
respectively.  The  problem  is  to  choose  a  control  law  ufc  so  as  to 
minimize  the  cost  functional  (2-3) . 

For  the  solution  of  the  optimal  control,  the  following  assump¬ 
tions  are  introduced  [29,31]: 

1)  Assumptions  of  the  theorem  2.1  are  satisfied. 

*  * 

2)  L(t,xt,ut)  -  xfc  Q(t,zt)xt  +  u  R(t,zfc)ut, 

where  for  n  e  Rm,  t  t  [0,T] ,  Q(t,n)  is  a  nonnegative 
definite  matrix,  and  R(t,n)  is  uniformly  positive 


definite, i.e. ,  elements  of  its  inverse  are  uniformly 
bounded  measurable  functions , . 

3)  The  control  ufc  e  U  satisfies 

/T  E [ II u  II2]  dt  <  ®, 

0  t 

ut  =  u(t,xt»zt) , 

and  is  such  that  (2-1)  has  a  unique  solution. 

Let  s  6  [0,T]  be  the  initial  time;  x  =  x_,  the  initial  state, 

s  0 

ufc  e  U,  and  xfc,  the  corresponding  response  of  the  system  (2-1) . 

The  conditional  remaining  cost  on  the  time  s  *  0  is  defined  by 

Wu(s,xg,zs)  =  E[/^  L(t,xt,zt,ut)dt  ]  xg  =  xQ,  zg  =  zQ] ,  (2-6) 

as  the  expected  cost  corresponding  to  the  control  ufc  and  initial 
state  xQ  and  zQ.  Here,  T  is  a  fixed  terminal  time  and  L{ is 
a  bounded  measurable  function.  The  problem  is  to  minimize  J(u)  on 

U. 

Let 

AfG(t,n)  G*(t,n)  G(t,n)  D  (t,n) 

G  (t  f  n)  — 

★  * 

.D(t,n)  G  ( t , ri >  D(t,n)  d  (t,n)  , 

and  assume  that  a(t,n)  is  uniformly  positive  definite  over  t  t  [0,T) 

„m 

n  e  R  ,  i.e. , 
n+m 

.1.  a  (t,n)y.y.  1  k]yy*|,  k  >  0, 

X'J  1J  1  i 

for  all  y  e  Rn+m.  This  essentially  states  that  noise  enters  every 
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Define 

V(t,y)  =  inf  VU(t,y) , 

where  VU(t,y)  =  WU(s,xt>zt)#  y  =[2tJ »  (t,y)  £  (0,T]  x  Rn+M. 
From  the  above  assumptions  it  follows  that  [Fleming  [4]] 

Vg  +  AU(s)V  +  L(s,y,  u (s  ,y) )  >_  0, 


where 


.u  a  i  ntm  a2 

A  (S)V  -  i  ,  L,  a,A 


2  i,3.l  -ij  3yl3y*  V  *  ill  <Als'T,,£ 
a  n+m 

+  B(s,n)us).  3—  v  +  i4fl(C(s,n))i  v,  n  £  R  , 

i 


5  e  R 


and 


V 

s 


A  3V 
*  3s  ' 


The  above  equality  holds  if  u  =  u°(s,y),  where  u°  is  an  optimal  feed 
back  control  law.  This  leads  to  the  continuous -time  dynamic-pro¬ 
gramming  equation: 


V  +  min  (  AU(s)V  +  L(s,y,u))  =  0,  V(T,y)  =  0. 
u 


(2-7) 


Theorem  2 . 2  Assume  that  the  value  function  V  satisfying  the 
stochastic  Bellman  equation  (2-7)  exists  and  is  differentiable  in 
(t,y).  If  a  control  u°  £  U  satisfies 

o 

Au  (t)  V  +  L(t,y,u°(s,y) )  <_  Au(t)  V  +  L(t,y,u(t,y) )  , 
for  all  ufc  £  U,  (t,y)  £  [0,T]  x  Rn  x  Rm, 


(2-8) 
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then  is  an  optimal  control. 


Proof  Let  ufce  U  be  any  control  and  y^  *  |  ^  j  the  corresponding 
trajectory.  From  Bellman's  principle  of  optimality  and  the  Taylor 
expansion  on  V, 


V(t,x  ,z  )  =  min  {L(t,x  ,z  ,u  )6  +  V(t,x . ,zj  +  XT  (t,x  ,z  )6 
ttueU  ttt  t  t  3t  tt 

t 

+  Au(t)  V(t,xt,zt)  +  0(6) }, 

where  any  6  e  (0,  T-t) .  Now  dividing  by  6  and  let  6-K),  then 


( t / x  / z  )  +  min  (Au(t) V(t,xt,zfc)  +  L(t,xfc,zt,ut)  *  0  (2-9) 

,t  utca 

V(T<Xt,zt)  =  0. 


Equation  (2-9)  is  the  Bellman-Hamilton-Jacobi ,  BHJ ,  equation  derived 
heuristically  above  (2-7) .  Then  from  (2-9) 


—  (t,xt,zt>  +  Au(t)V(t,xt,zt)  +  L(t,xt,zt,ut)  >_  0. 


Thus,  using  Ito's  formula  it  follows  that 


££  (V(t,xt,zt))  -  (t,xfc,zt)  +  Au(t)  V(t,xt,zt)  >_ 

-L(t,xt,zt,ut) . 


(2-10) 


Taking  the  expectation  of  the  both  sides  of  (2-10)  and  integrating, 
the  following  equation  is  obtained. 


T  dV 

E[V(T,xT,zT)  -  V(0,xQ,zo))  =  Ei;0  ^  (t,xt,zt)dt 


(2-11) 


>_-  E/q  L(t,xt>zt,ut)dt. 


Since  V(T,xT,zT)  *  0,  this  shows  that  the  following  inequality  holds: 
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E[V(0,x  ,z  )]  <E  /T  L(t,x  ,z  ,u  )dt  =  J(u  ).  (2-12) 

0  0  ~  0  ttt  t 

With  u  °.  the  same  calculations  apply  to  give 

E  [V(o,xq,zq  )  ]  =  J(ut°)  (2-13) 

But  now  (2-11)  and  (2-13)  show  that  u  °  is  optimal;  hence  the  above 
results  imply  that 

j(ut°)  £  J(ufc) ,  tot  all  ufc e  U.  ■  (2-14) 

Assume  that  the  value  function  V(t,xt>zt)  on  [0,T]  x  R  x  R 
has  the  following  from  [29,31]; 

V(t,xt,zt)  =  xfc*  A1(t,zt.)xt  +  A2(t,zt)xt  +  A3(t,zt),  (2-15) 


where  A  ,  A2<  and  are  the  solutions  of  certain  nonlinear  partial 
differential  equations  and  xfc  e  Rn,  zfc  e  Rm.  Then,  the  stochastic 
BHJ  equation  is  given  by 


_3_ 

3t 


Al(t,2t)Xt  +  A2(t,zt)Xt  +  A3(t'zt)) 

+  min(AU(t)  (xt*  A1(t,zfc)xt  + 
u  tell 

+  A3(t,zt))+  L(t,xt,zfc,ut) )  *  0  . 


(2-16) 


It  is  enough  to  show  that  we  can  find  A^,  Aj  an<^  ut°  that 
satisfies  (2.16).  This  is  shown  in  the  theorem  below. 


Theorem  2 . 3 


The  optimal  control  ufc 


t  e  [0,T] ,  exists  and  is  given  by 


~  '■  '  £*—■**■' 


o 
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ut°  =  -R  1(t,zt)B  (t,zt) (A1(t,2t)xfc  +  i  A2(t,zt) ) , 


(2-17) 


if  there  exist  the  nonnegative  definite  symmetric  matrices  A  and 
satisfying  the  following  nonlinear  partial  differential  equations  : 


A,  +  A*A  +  A, A  +  Q  -  A  BR-1B*A,  +  C*~—  A, 

1  111  1  3z  1 

2  t 

+  itr(DD*ij^*)Ai  =  0' 


(2-18) 


K  *  V  -  V’rVi'l  *  C*sl;  *2  +  I 

*  3  * 

+  2  (GD  -r - )  A  =  0, 

3zt  1 

Ai(T,zt)  *  o,  A  (T.z  )  =0,  z  e  Rm, 

(the  argument  (t,zfc)  is  omitted  for  bervity) . 


Proof.  Using  the  dynamic-programming  equation  (2-16) ,  it 


follows  that 


*3A  3A  3A  n  n  .  . 

Xt  lt“  Xt  +  3t“  Xt  +  3t^ 

n+m  g  g  n+m  n  3  3 

+j*n+iaij ^ 3xfc^ i  ^3x^j^  +  i-n+l^j^l  °i j ^ 3z^ i ' 3x^ j 


3  3  * 

+  .  Z  c.  .  (q-—)  •  (^ — ).))-(x„  A, xk  +  Ax  +  A,)  (2-19) 


r  .  _  ,  5  ‘v  .  .  \  rv  /  .  V  ^  /  .  /  /  •  \  ^  .  ‘V-  «  ■  A  t-  / 

3«n+l  ij  3zt  i  3z  ]  tit  2  t  3 


♦  i-i<Axt  *  *  Vt  *  v 


*  JilcV3i;,i,xt\\  *  Vt  *  V  *  \*0\  *  ut\>  - 


The  equation  (2-19)  can  be  represented  by  the  following  equation: 
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x  Ax  +  A  x  +  A  +  ^(  ( .  ?  o  .  .  (-r^- )  .  (t~“)  .  + 

tit  2  t  3  2  1,3=1  13  3x  l  3xt  3 


“  “  a  3 

I  <3  ..{■£-).  (T2-)  .)  •  (X.  Ax  +  Ax  +  A  )  + 

.  13  3z.  1  3zfc  3  tit  2  t  3 

i,3=n+l  t  t  J 

n  n+m  .  .  n+ra  n  3  3 

(  E  I  a.  .  ("r— )  .  (?£-)  .  +  I  £  °,  ,  (tt-)  .  (3—)  . ) 

i=l  3=n+l  J  t  t  J  i=n+l  3=1  t  t  J 

*  m  3  *  * 

•  (x  A  x  +A  x  ) )  +  Z  (C)  .  (-r— -)  .  (x  A  x  +  A  )  +  x  Qx 
tit  2  t  .  ,  1  3z  1  t  1  t  3  t  t 

1=1  t 

^  3  ^  * 

+  min (  I  (Bu  ) .  (t — )  .  (x  Ax  +  Ax.)  +  u  Ru  )  =  0  .  (2-20) 

,  t  1  3x.  ltlt  2  t  t  t 
u  eU  1=1  t 


Only  the  last  term  in  (2-20)  includes  the  control,  and 


2  (But)i(3T  (xt\(t'Zt)Xt  +  A2(t'Zt)Xt)  +  Ut*R(t'2t)Ut) 
i=l  t 

=  (ut  +  R-1 (t,zfc) (B  (t,zfc) A1(t,zt)xfc  +  ^  B  (t,zfc)A2(ttzt) )) 
•R(t,zt) (ut  +  R_1(t,zt) (B  (t-zt) A1(t,Zt)xt  +  -J  B  (t,Zfc) A2 (t,zfc) ) ) 

-  (R_1(t,zt) (B*(t,z  )A  (t,*t)xt  +  |  B  (t,zt)A2(t,zt))* 

•  R  1 (t » z^) (B  (t,zt)Ai(t,zt)xt  +  ~  B  (t,zt)A2(t,zt)) .  (2-21) 


It  follows  that  the  minimum  is  obtained  at 


ut°=  -R  1 (t,zt)B  (t,zt) (A1 (t,zfc)xt  +  ~  A2  (t,zt) ) . 


(2-22) 


Substituting  (2-22)  into  (2-20)  gives 


t  at  1  1  T 


3z  1  2 


-  A1B»-1B*A1)xt  *  (|jA2  +  A2A  *  C*^-  A2  *  }  tr  A2 

*  2l00*»^»*Al  -  +  <it  S  *  °*ai;  aa 


+  4  tr  ( DD*t — — r~ — y)  A  +  tr  (GG*) A. )  -  0. 

2  dZ^oZ^  3  -»■ 


(2-23 


Because  (2-23)  has  to  be  established  for  all  xte  R  ,  h  ,  A^»  A^ 
must  satisfy 


Vil  _  ^  4  ^  ^  ^ 

5T  AjA  -  a  *  *!»-  B  A2  -  C  jj-  A,  - 

1  *  a2 

2  te^7>Al- 

ir  ■  -A2a  *  A2BR'V\  -  ^  a2  -  i  “®D*T5^7|A2  - 

*  3  * 

2(0D^-)  A2. 

3A3  *  3  1  *  92  * 

3T  '  -C  IT  A3  -  2  tt(DD  30i7>A3  -  trl“  >‘V 

t:  t  t 


Ai<T,zt)  =  0,  A2(T,zt)  -  0,  A3(T,zt)  *  0, 


respectively. 


The  solutions  A  (t,n)  and  A2(t,n)  to  the  above  Cauchy  problem  can 
be  shown  to  be  nonnegative  definite  and  uniformly  bounded  for  all 


(t,ri)  £  iO,T]  x  R  [29]  . 


Mohler  developed  bilinear  stochastic  systems  that  are  the 
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diffusion  models  for  migration  of  people,  biological  cells,  etc., 
[30,31,66,67,68].  The  system  equation  in  (2-1)  is  a  class  of 
coupled  bilinear  stochastic  equations.  In  this  particular  case, 
the  optimal  control  of  the  bilinear  stochastic  system  of  diffusion 
processes  (2-1)  and  (2-2)  is  given  by  (2-17) .  The  following  examples 
belong  to  the  class  of  coupled  bilinear  stochastic  systems. 

Example  2.1  If  A(t,zt)  =  0,  G(t,zt)  =  G(t),  then  the  equation 
(2-1)  is 


dx  =  B(t,-)u  dt  +  G(t)dw  ,  x(o)  =  x  e  R  ■ 
t  t  t  0 


(2-24) 


where  B(t,-)  is  composed  of  unknown  coefficients.  Such  uncertain 
parameters  may  be  regarded  as  additional  state  variables.  These 
additional  state  variables  with  uncertain  gain  might  be  approximated 


dz.  *  C(t,z  )dt  +  D (t)dw  ,  z(0)  =  z  SB. 
t  t  to 


(2-25) 


If  B(t,')  =  B (t) z  ,  (2-24)  is  bilinear  in  z  and  u  ,  and  the  system 
^  t  t 

has  an  extended  state  with  Rn+m.  At  this  point  the  problem  of  un¬ 
certain  parameter  becomes  a  parameter-identification  problem  and 
the  system  equation  is  a  bilinear  stochastic  differential  equation. 
An  aircraft  landing  process  [44]  may  be  represented  by  this  type  of 
bilinear  stochastic  equation. 


Consider  the  stochastic  differential  equation  with  the  random 


coefficients, 


S 

L 
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dxfc  =  A(t,zfc)  xt.  dt  +  B(t,zt)  u  dt  +  G(t)dwt1,  (2-26) 

where  state  xfc  e  Rn  is  observable  and  uncertain  disturbance  process 
zfc  e  Rm  is  partially  observable;  and 

2 

dz^  =  C(t)z^_  dt  +  D(t)  dw^ 

with  the  observation 

dyt  =  F(t)zfc  dt  +  H (t)  dw^. 

The  problem  of  optimal  control  of  (2-26)  under  the  given  information 
will  be  discussed  in  Chapter  3. 

Comment:  The  optimal  control  of  partially  observable  state  xt  was 

discussed  by  Kolodziej  [29] -  There  exists  the  optimal  filter  in 

the  conditionally  Gaussian  case.  The  form  of  optimal  control  law 

of  this  particular  case  is  the  same  as  the  equation  (2-17). 

The  stochastic  dynamic  model  in  (2-26)  may  be  approximated  in 

certain  ways  for  0-algebra  V  generated  by  observation  (y^.t  e  [ 0 , T] } 

or  o-algebra  generated  by  observation  {xfc,y  ,t e  [0,T]}.  If  there 

is  an  exact  solution  to  the  problem  of  finite  conditional  estimate 

E(  | Z. ] ,  the  stochastic  dynamic  equation  may  provide  optimal  control 
zt  C 

of  (2-26).  If  A(t,zt)  and  B(t,zfc)  in  (2-26)  are  replaced  with 
E[A(t,zt) ] y  ] ,  E[B (t,zfc) ( y  ] , .  the  problem  of  sub-optimal  control 
is  similar  to  that  introduced  at  the  beginning  of  this  chapter. 
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3.  APPROXIMATE  STOCHASTIC  MODELS 

Formulating  a  mathematical  stochastic  model  for  the  dynamic 
behavior  of  a  physical  system  is  in  terms  of  the  evolution  of  the 
state  xfc,  t  e  [0,T] ,  of  the  system  which  is  a  stochastic  process 
defined  on  some  complete  probability  space  (C,F,P)  under  the  in¬ 
fluence  of  control  ufc  and  disturbances  z  as  the  solution  of  the 
stochastic  differential  equation, 

(3-1) 

dxfc  =  A (t, z^) x^_dt  +  B(t,zt)utdt  +  G(t,zt)dwt1,  x(0)  =  xq, 

and  xfc  is  an  observable  process  and  w^1  is  a  Wiener  process.  Assume 
that  this  stochastic  system  has  disturbances  as  the  solution  of  a 
stochastic  differential  equation 

dzt  =  C(t)zt  dt  +  D(t)dwt2  ,  z(0)  «  z  ,  (3-2) 

with  observation 

dyfc  =  F(t)z  dt  +  Httjdw^3,  (3-3) 

where  z  is  unobservable  and  w^.1 ,  i  =  1,2,3,  are  mutually  independ¬ 
ent  Wiener  processes  of  dimensions  ,  respectively.  The  matrices 
A,  B,  G,  C,  D,  F ,  and  H  have  the  dimensions  n  x  n,  n  x  p,  n  x  l  , 
m  x  m,  m  x  t 2»  k  x  m  and  k  x  ty  respectively.  Assume  that  A(t,zt) , 
B(t,zt>,  and  G(t,zt)  of  ( t,z c  [0,T]  x  Rm  are  Borel  measurable. 

Let  be  the  a-algebra  generated  by  {xg  and  yg,  0  s  <_  t}. 

The  control  ufc  of  dimension  P  is  assumed  to  be  measurable  for 
every  t  t  [0,T] .  The  control  u  is  to  be  chosen  so  as  to  minimize 


the  cost 
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J(u)  =  E[/q  x  Q (t , zfc) xfc  +  ufc  R(t,zt)ut]dt, 


(3-4) 


where  the  symmetric  matrices  Q  and  R  have  the  dimensions  n  x  n, 
and  p  x  p,  respectively. 

Consider  the  mean-square  estimate  of  z  :  E[z  jy  ,  0  t  <_  T]  . 

y.  denotes  the  a-algebra  generated  by  {y  ,  0  s  <_  t}.  Let  z  be 
t  st 

E[zt|yt,  0  <_  t  <_T]  ,  and  V  C  Z  .  Under  the  proper  assumptions,  the 
estimate  zfc  satisfies  the  linear  stochastic  equation  given  by 


dz  =  C(t)  ztdt  +  F ( t) * (  H (t)  H(t)*)-1d  v  , 


z0  =  E[zq], 


(3-5) 


where  dv^  is  the  innovation  process  corresponding  to  (3-3) ,  and 
Ft  is  the  error-covariance  matrix  which  satisfies  the  following 
matrix  Riccati  equation: 

dr  =  (d (t) d* (t)  -  r  F(t)  (H(t)H(t)*)"1  F(t)r  +  c(t)r  + 

^  II  t  t 

rtC*(t))dt»  (3-6) 

r0  "  Cov[z0l. 


The  system  of  equations  (3-5)  and  (3-6)  have  a  unique  solution  for 
r  in  the  class  of  symmetric  nonnegative-definite  matrices. 

3.1  Simple  Approximation 
to  Stochastic  Modeling 


The  stochastic  system  model  in  (3-1)  has  an  unknown  random 
coefficient  which  depends  on  the  unobservable  disturbance  stochastic 
equation  (3-2) .  For  the  first  method  consider  the  approximate 
stochastic  model  for  (3-1)  . 


I 


dx^  A(t,zt,rt)  xt  dt  +  B(t,zt,rt)  ut  dt  +  G(t,zt,rt)d  , 


where 


A(t,£,r  )  =  o  <  t£T] 

^  U  21 

=  £  A(t,£)-  (l/(2ir)2!"t|i)  •EXP[-|(£-£)*rt_1(£-£)]d£, 

_  „  (3-8) 

B(t,£,rt)  =  E[B(t,£)  |Vt.  0  it<T], 

G(t,£,rt)  =  E[G(t,£) \Vt.  0  <  t  <Tl  . 


Here  the  mean  estimate  zfc  is  the  solution  of  (3-5)  and  the  covariance 
r  is  the  solution  of  (3-6) .  The  problem  is  to  find  the  control  ufc 
from  the  admissible  class  that  minimizes  the  cost  functions  (3-4). 

For  the  linear- regulator  problem,  (3-4)  is  approximated  by  the 
following  new  cost  functions: 

J(u)  =  E[/T(x  Q(t,z  ,r  )x  +  u  R(t,z  ,r  )u  )dt] ,  (3-9) 

O  t  t  t  t  ^  w  L  t 


where 


2(t,£t,rt)  =  /pm  Q(t,£t) -f(£t,rt,£t)d£t, 
R(t,?t,rt)=  ;Rm  R(t,£t)  •f(Ct>rt,£t)d£t. 


(3-10) 


A  m 

Here  f(£t,r  ,£  ),  £fc  £  R  ,  t  z  [0,T] ,  is  the  m-dimensional  Guassian 
density  function  with  mean  £  and  covariance  This  transforma¬ 

tion  results  in  the  case  of  the  system  (3-7),  and  the  symmetric 
matrices  Q  and  R  have  the  dimensions  n  x  n  and  p  x  p,  respectively. 

Remark:  The  distribution  of  the  estimation  error  £  *  £  -  £  is 
Gaussian,  and  Q  (t,£,ft)  and  R(t,£,ft)  may  be  calculated  by  (3-10). 
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J(u)  has  the  similar  form  of  J(u). 

Let  U  be  a  certain  class  of  admissible  controls.  The  control 
ut°£  U  is  called  optimal  for  (3-7)  if 

J(u.°)  *  inf  J(u  ),  (3-11) 

u teU 

where  inf  is  taken  over  the  class  of  all  admissible  controls.  One 
of  the  analyses  of  the  above  control  problem  is  included  to  get  the 
optimal-control  law  of  the  observable  linear  control  system  with 
quadratic  criteria  which  has  random  coefficients  being  certain 
functionals  of  the  Wiener  process  v  [19]  . 

For  the  solution  of  the  optimal  control  of  (3-7) ,  it  will  have 
the  same  assumptions  as  are  made  in  Chapter  2  with  the  proper 
parameters  of  (3-7) .  Then,  the  unique  strong  solution  of  (3-7) 
exists  because  the  control  ufc,  0  _<  t  £  T,  is  admissible  if  for  this 
control,  theorem  2.1  and  assumptions  1 ) —  3 )  in  Chapter  2  is  satisfied. 

Assume  that  the  value  function  is  of  the  following  form 


V(t,xt,£,Y)  *  xfc  A  (t,£,Y)xt  +  A2(t,£,Y)xt  +  A3(t,£,Y),  (3-12) 

—  ni  _m  n\ 

xfce  R,£eR,Y£R  xR, 


where  A^f  A and  A^  are  symmetric  matrices  which  satisfy  a  certain 
nonlinear  partial  differential  equations  which  will  be  discussed  later. 
The  assumed  ,y)  is  some  smooth  function. 

Comment : 

The  stochastic  integral  might  be  defined  by  a  stochastic  in¬ 
tegration  in  the  mean-square  sense  of  Ito  or  Stratonovich.  The  Ito  in¬ 
tegral  is  much  easier  for  computation  of  expectation  of  the  Ito  integral 


J 
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than  the  Stratonovich  integral,  and  it  has  other  nice  mathematical 
properties  not  possessed  by  the  Stratonovich  integral.  On  the 
other  hand,  the  Ito  stochastic  differential  rule  as  given  by  the 
following  theorem  states  the  conditions  where  by  a  certain  random 
process  is  permitted  a  stochastic  differential  [64] . 

Theorem  3.1 

Let  the  function  f(t,xt,6,Y)  be  a  measurable  smooth  function 

which  has  partial  derivatives  f  ,  f  ,  fr,  f  ,  f  ,  f,r,  f„  » 

t  6  -i  xtxt  U  xtr 

f  ,  f  .  The  Ito  formula  is  then  given  by 
CY  YY 


(3-13) 


d  f(t,xt,5,Y)  =  f  (t,x  ,C,Y)dt  +  fx  (t,xt,£,y)cbct 

+  f^(t,xfc,C/Y)dC  +  f  ^(t,xfc,C,Y)<3  Y  + 

if  (t,x  ,4,Y)GG*dt 
i  xtxt 

+  Tf„(t,x,,5,Y)  K(t)  K  (t)  *dt, 

2  t 

where  K(t)  =  yF(t)*(H(t)  H(t)*)"1. 


Proof:  Omitted  (see  [64]). 


These  stochastic  integrals  suggest  that  the  correct  formula 

for  df(t,x  ,£,y)  is (3-13)  where  G  and  K  stem  from  parameters  of 
*  *  -1 

(3-7)  andTtF(t)  (H(t)H(t)  )  ,  respectively. 

Using  theorem  3.1,  the  differential  form  of  the  value  function 
(3-12)  is  given  by 

*3A1  8A2  -  * 
dV(t,xt,S,Y)  =  xfc  xtdt  +  xt  dt  +  2(Axt  +  But>  A^dt 

+  (Axt  +  5ut)  A2dt  +  tr  (GG  A^dt 
+  (FC)*^  +  A2X  t+  A3)U=z[dt 

(cont. ) 
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+  (D(t)D(t)  -  YF (t)  (H(t)H(t)  )  F(t)Y 

+  C(t)Y+  YC(t*))*  .  (-^T  (xfc  A1xfc  +  (3-14) 

A2  xt  +  A3)!r=rt>dt 

+  |  tr  *t  +  Vt  +  A3)'^))dt 

*  -  1 

+  (2xt  A  +  A2)G  dwt 

+  (-^(xt*Aixt +  Vt  +  V1  ?=zt)KdV 


Taking  the  integral  of  both  sides  of  (3-14) ,  it  follows  that 

v(T,xT,2T,rT)  -  V(0,x0,z0,r0>  =  fTo  ut\uA±)  +  2A*A1)xt 

+  2(iut)*xfc  +  L(A2)  +  A*A25xt  +  tr(5c  )A1 

T 

+  (But)*A2  +  L(A3))dt  +  fQ  ((2  xt*Ax  +  A2)G  dwA 


jf'SVt  +  Vt  *  A3lE-it,MVt’' 


(3-15) 


where 


L(.)  -•£<•)♦  UFV'sT  <•>  +ltt(rac*HkT(-,,,1C.it 


it  *  It  mm  1  *^9  I 

+  (DD  -  yF  (HH  )  FY  +  CY+  YC  )  (*) lY.r  ' 

and  all  arguments  (t,C,Y)  are  omitted  for  brevity.  The  equation  of 
(3-15)  is  given  by 


E(v(T,xT,zT,rT) 


V  (0,x0,.0,r0)) 

E(  /T  x'  (LI*,)  +  »**  +  *,»)«. 

0  t  1  lit 

+  2(But)*xt  +  <L(A2)  +  A*A2)*t 
+  tr(GG*)A  +  (But>  A2  +  L(A3))dt. 


(3-17) 


r  ****■*#» 
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Consider  the  formal  application  of  Bellman's  principle  of  optimality 
along  with  the  differential  formula  which  suggests  that  V  should 
satisfy  the  stochastic  Bellman  equation. 

Theorem  3.2 

The  optimal  control  in  the  approximation  stochasitc  system 
(3-7)  is  given  by 

(3-18) 

u  °  =  -R_1  (t,zt,rt)  5  (t,zt,rt)  (A1(t,zt>rt)xt  +  |  /2(t,zt,rt) ), 

where  A^trZ^,^)  and  A2(t,zt,rt)  satisfy  the  following  Riccati-like 
equations : 

2 

K  =  -  V  +  Q  -  A1BR-15*A1  -  (FC)*|?A1-  |  tr(KK*^TA2) 

-(DD*-  YF*(HH*)_1FY  +  CY  +  YC*)*  A^  (3-19) 

A 2  =  _A2A  "  A2^  ®  A1  "  3TA1  J  tr(KK 

-(DD*-YF*(HH*)_1FY+  CY+  YC*)*|^  A2 . 

The  arguments  (t,C.Y)  are  omitted  for  brevity,  and  A1(T,C,y)  =  0, 
and  A2<T,C»Y)  =  0. 

Proof : 

Under  the  same  consideration  of  theorem  2.4  in  Chapter  2,  the 
stochastic  Bellman  equation  is 

*  —  *  —  *  —  — 

Xt  ^L^Ai^  +  A  +  AiA)xt  +  xt  ®Xt  +  L(A2)xt  +  A2AXt 
_ * 

+  L(A  )  +  tr(GG  A  )  +  min  (u  Ru  +  2u  B  Ax 

3  1  xl  1  Z 

+  ut*B*A2)  =  0.  (3-20) 

Note  that  the  last  term  of  (3-20)  only  depends  on  ufc. 


Hence  , 
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ut*£ut  +  2ut*i*A1xt  +  ufc*B*A2 


(ut  +  R"1(B*A1xt  +  |  B*A2)*R(ut  +  R~1(B*A1xt 

*i5  V> 

(R~1{B*A1xt  +  ~  B*A2)  (R_1(B*A1xt+  ~  B  A2)). 


Thus,  the  minimum  is  achieved  at 


ut°  -  -R_1  i‘(Axxt  +  \  A2) 


(3-21) 


(3-20)  is  substituted  by  (3-21)  ,  and  then 


xt  (L (A^)  +  A  A2  +  A^A)xt  +  xfc  Q  xfc  +  A2  Axfc  +  L(A2)xfc  +  LtA^) 

_ *  *  _ 2-*  _ 1-* 

+  tr(GG  A^)  -  x^  A^BR  B  A^x^  -  A2BR  B  A^x^  =  0. 

Therefore,  (3-22)  is  the  solution  as  long  as  A^,  A  ,  and  (3-22) 

A^  satisfy 

3A_  ^  ^  „  it  'a 

-T— ■  +  A  A  +  A.  A  +  Q  -  A  BR_i  BA.  +  (Fz.)  -r-  A. 
dt  1  1  1  It  1 

+  (dd*-  rt  *(HH*)-1Frt  +  crt  +  rtc*)*  g|-  ai 
2 

-  tr  (KK  3r  ^3r  A1)  =  0, 

^A^  1  ^  A  ^  S 

+  A0A  -  A  5r~  B  A  +  (Fz  )  -7-  A 
dt  2  2  It  3zt  2 

+  (dd*  -  rtF*(HH*)_1Frt  +  crt  +  rt  cV~-  a2 

2  ^ 

-  -  tr (KK  3r^3r^*  A2}  =  °' 


^  ^  A  ^  ^  A  1 

3t  +  (FZt}  3z  A3  +  <DD  “  FtF  (HH  Frt  +  CFt 

*  rtC*,#3T“  A3  -  7  tr,KK’8r5I7  V  +  tr(5='V  -  »• 

t  t  t 

a, (T,z_,r  )  =  o,  A„(T,z.r  )  =  o,  a  (t,z  , r  )  =  o.  ■ 
ITT  2TT  3Tt 
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The  above  results  for  BHJ  equation  have  the  following  inequality: 

J(u  °)  <  J(ut)  ,  ufc  e  U. 

The  control  ufc°  defined  by  (3-18)  is  admissible  since  the  stochastic 
equation  (3-7)  has  a  unique  strong  solution. 

Now  take  into  account  the  stochastic  system  (3-1)  with  random 
coefficients.  The  different  approximation  method  has  also  been 
modified  by  the  proper  model  using  the  mean  estimate  of  (3-2) , 

dx^  s  A(t,zt)xtdt  +  B(t,zt)utdt  +  GCtjZ^dw^1,  (3-23) 

x(0)  =  Xq . 

Approximation  model  (3-23)  is  simpler  than  (3-7),  the  first  ap¬ 
proximation  to  stochastic  system  equation  (3-1).  If  A(t,zt)  does 
not  have  the  form  of  A(t)zt  in  (3-7)  and  (3-23), 

E[A(t,zt) \Vt.  0  <  t  £  T]  jt  A(t,zfc) . 

Therefore,  the  equations  (3-7)  and  (3-23)  are  different  approximation 
models,  in  general. 

Consider  the  problem  of  optimal  control  of  stochastic  model 
(3-23)  with  the  cost  function  (3-4).  In  this  case  the  problem 
has  the  modified  cost  function  which  is  the  same  as  (3-9).  Again, 
define  the  following  value  function  corresponding  to  (3-23)  as 

V'(t,xfc,C)  =  xt  A'(t,£)xt  +  A'(t,£)xt  +  A'(t,£) , 

where  A',  A',  and  A'  are  symmetric  matrices  which  satisfy  certain 
nonlinear  partial  differential  quations. 

note  that  strong  and  weak  solutions  are  discussed  in  [64} . 
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Theorem  3 . 3 


Let  A'  and  A^  be  the  bounded  symmetric  solution  of  the  follow¬ 


ing  Riccati-like  equation: 


*  -  — 1  * 

A  A'  -  A1A  +  ®  -  A'BR  B  A' 

1  *  g2 

-  -  tr(KK  J^r  A1)  , 


A; 


Itr(KX*3WA2>’ 


where  the  arguments  (t,£)  e  [0,T]  x  R  are  omitted  for  brevity, 
and  A^  (T,5)  =0,  A'(T,C)  =0,  for  £  e  R  .  Then,  the  optimal 
control  of  (3-23)  under  proper  assumptions  is  given  by 

u  °  =  -R  1(t,z  )B  (t,z  ) (A"(t,z  )x  +  \  A'(t,z  )).  (3-24) 
t  t  t  1  t  t  2  2  t 


Proof : 


The  proof  is  similar  to  theorem  3.2.  ■ 

Remark:  The  optimal  controls  of  (3-7)  and  (3-23)  are  suboptimal 

for  (3-1)  because  the  class  of  stochastic  models  of  (3-7)  and 

(3-23)  are  approximation  models  of  (3-1)  using  E[zt|/  ,  0  <_t  ^_T  ] 

and  E[  (z  -z  ) 2 1  V  ,  0  <_  t  <_T]  . 
o  t  t 


3.2  Suboptimal  Control  of  a  Class  of 
Stochastic  System  with  Unobservable 
Random  Parameters 


In  general,  uncertain  unobservable  stochastic  processes  in 
(3-2)  have  nonlinear  observation  equations;  then,  the  optimal 


mean-square  estimate  of  unobservable  states  is  the  solution  of  the 
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certain  infinite-dimensional  equations.  Hence,  it  is  necessary 
to  model  an  implementable  approximation  and  evaluate  its  perform¬ 
ance  in  (3-1).  Let  xfc,  t  e  [0,T] ,  satisfy  the  following  Ito-type 
stochastic  equation: 

dxt  =  A(t,2fc)xtdt  +  B (t, 2fc) u^dt  +  G(t,zt)aw^  ,  (3-25) 

where  z^  is  an  unobservable  stochastic  process  satisfying 

dzfc  =  C(t,zt)dt  +  D(t,zt)dwt2.  (3-26) 

Assume  that  xfc  and  y^_  are  observed,  where  y  satisfies 

dy,.  =  F(t,zt)dt.  +  H(t)dwfc2.  (3-2”' 

1  2 

Processes  x^,  z^,  Yfc,wt,  wt  are  of  dimensions  n,  m,  t  ,  q^,  q  ,  re¬ 
spectively,  and  all  matrix  functionals  A,  B,  G,  C,  D,  F  are  of 

1  2 

appropriate  dimensions.  It  is  also  assumed  that  w^  and  w^_  are 
mutually  independent.  Each  of  the  measurable  functionals  A(t,9) , 

B (t, 0) ,  G(t,0),  C(t,0),  D (t, 0) ,  F ( t , 0)  is  assumed  to  be  non- 
anticipative.  The  p-dimensional  stochastic  process  u^,  referred  to 
here  as  a  control,  is  assumed  to  be  ^-measurable,  where  is  the 
a-algebra  generated  by  (xg,ys,  0  <_  s  £  t}_ 

The  problem  is  to  find  the  control  that  minimizes  the 
cost  functional, 

T 

J(u)  *  Ef  L(t,x  ,u  )dt.  (3-28) 

o  t  t 

(3-25),  (3-26),  (3-27)  may  be  interpreted  as  a  linear  stochastic 
control  system  with  random,  partially-observable  parameters.  If 
the  stochastic  differential  equations  (3-25),  (3-26),  (3-27)  are 
satisfied  in  theorem  2.1  in  Chapter  2,  there  exists  a  unique  solu¬ 
tion  of  (3-25),  (3-26),  (3-27),  respectively. 


Let  E  [.]  =  E[- |y  ]  where  V  is  the  a-algebra  generated  by 
{y  ,  o<_s  <t}.  The  following  result  is  now  needed: 

Theorem  3.4.  If  f  is  a  c2~class  function  and  satisfies 


/  Elf (z  ) (HH  )  ’  F(t,z  ) |  dt  <  «, 
0  t  t 


then. 


Et[f(zt)]  =  E(f(zfc)|ytl 


satisfies  the  following  stochastic  differential  equation: 


dEt[f(*t>]  =  Et[Cf(zt)]dt  +  {Efc[f (zt)F(t,zt)] 

-  E  [f(z  )]EttF(t,z  )]  +  H(t)Et[D]} 

(H(t)H(t))  *(dyfc  -  Et[F(t,zt)]dt) , 


(3-29) 


where 


Cf  =  ^  C(t,zt).(^)i  +  J  j  )ij<ai^)ij' 


i»  j=l 


(3-30) 


”, 

1=1  :i  3zfc  j 


(3-31) 


Proof :  Proof  can  be  found  in  [69],  Here,  if  f(zfc)  =  zfc,  (3-29) 

* 

is  the  conditional  first  movement,  and  if  f(zfc)  =  z^z^  ,  (3-29) 
is  the  conditional  second  moment. 


Assume  now  that  m  *  1  (zfc  is  a  scalar  process)  and  that 
N 

f(zfc)  =  z^,  where  N  is  a  positive  integer  number. 
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Then,  in  general,  the  stochastic  differential  equation  (3-25) , 

(3-26) ,  (3-27)  and  performance  index  £3-28)  do  not  form  a  closed 

N 

system  as  the  equation  for  d£t[ztl  involves  the  next  higher  con¬ 
ditional  moment.  This  means  that  solution  to  an  infinite  di¬ 
mensional  set  of  differential  equations  may  be  required  in  order 
to  obtain  conditional  moments  of  z^.  However,  further  assumptions 
are  made  here  so  that  either  of  the  following  situations  occur: 


1)  The  above  equations  form  a  finite-dimensional  set, 
i.e.,  there  exist  K  >  1  such  that  for  all  N  >  K 

E  (z^)  =  function  (E  (z  ),  E  (z  2) ,  ...,  E  (z  K) ) ;  or 

2)  an  approximation  technique  is  used  to  obtain  a  finite¬ 
dimensional  set  of  equations  for  the  Kth-conditional 
moment  of  zfc. 

The  information  available  to  the  controller  allows  for  a  better 
mean-square  estimate  of  zfc  which  is  given  by  z t  since  xfc  contains 
information  about  zfc  as  well  as  yfc.  Here,  it  is  assumed  that 
the  estimator  zfc  is  suboptimal  in  the  sense  that  only  the  obser¬ 
vation  yg,  s  e  [0,T],  is  used  to  construct  the  estimate.  These 
assumptions  decouple  the  control  problem  and  estimation  of  zfc 
problem. 

In  order  to  solve  the  suboptimal  minimization  problem  stated 
by  (3-25)  and  (3-28),  the  following  approximation  of  (3-25)  is 
used: 


dxt  . 


x. dt  +  B^U  dt  +  G  dw^ , 
t  t  t  t  t 


(3-32) 


where 
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At  =  E [A (t,zfc) 1 yfc3 / 

it  -  E[B(t,*t) { y  1  ,  (3-33) 

Gt  =  E[G(t,2t)  | y  ] . 


The  above  assumption  transforms  the  partially-observable  stochastic 
problem  into  a  completely  observable  one.  The  tools  for  solving 
the  above  approximate  control  problem  are  still  limited.  Here, 
one  may  refer  to  [16],  where  a  similar  structure  of  controlled 
diffusion  process  is  discussed  for  quadratic  form  of  the  cost 
function  L.  Note  that  in  this  case,  a  different  approximation 
is  used.  It  is  assumed  that  Afc,  B^,  G  can  be  approximated  or 
described  exactly  by  a  measurable  function  of  the  conditional 
moments  ^t[2t  ],  N  =  1,  2,  . . . ,  K,  ...  .  With  the  assumptions 
made  previously  about  finite  dimensionality  of  the  suboptimal 
filter  for  the  estimator  of  z^_,  it  follows  that 

A^_  =  A  (t ,  )  , 

Bt  3  B(t,5t> ,  (3-34) 

Gt  s  G(t,C  ), 


where 

5.  =  (E  (z  ) ,  E  (z  2) , . . .  ,.E  (z  K) ) , 

t  t  t  t  t  t  t 

satisfies  the  set  of  equations  of  the  form 


(3-35) 


d£  -  S(t,e.)dt+  P(t,£.)dv  . 
t  t  t  t 


(3-36) 
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Now,  the  approximate,  completely  observable  version  of  (3-25) , 
(3-26) ,  (3-27)  takes  the  form  of 

dxfc  =  A(t,Ct)xfcdt  +  B(t,C  )ufcdt  +  Gtt.S^dw^1,  (3-37) 

d?  =  S(t,5t)dt  +  P(t,Ct)dvt, 

where  is  a  Wiener  process  independent  of  w^1  because  of  the 
independence  of  w  ^  and  w^. 

If  the  cost  function  J(u)  in  (3-28)  is  of  a  quadratic  form, 

i.e. , 

T  # 

J(u)  =  E(/  (xt  Q(t)xt  +  ut  R(t)u^_)dt  +  xT  M  xT]  .  (3-38) 

The  following  results  apply  (29] . 

Let  the  following  assumptions  be  satisfied  for  all  t  £  [0,T] , 
n  e  Rk: 

1)  ||  A < t , n ) I!  +  1  B(t,n)||  +  ||  G(t, n)|j  <  k  <  », 
where  k  is  a  finite  positive  contant ; 

2)  S(t,n)»  p(t,n)  are  such  that  a  unique  strong  solution 

of  (3-46)  exists  and 
T  7 

Prob.  (/  3  ^dt<  »  )  =  l; 

3)  Q(t)  and  M  are  non-negative  definite,  and  R(t)  is  uni¬ 
formly  positive  definite  (i.e.,  its  inverse  is  uniformly 
bounded) ,  and 

4)  u^  satisfies 

T  ? 

f  0  H(||ut  ||)dt  £ 
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Theorem  3 . 5  Under  the  above  assumptions,  if  there  exists  a 
bounded  solution  V(t,h)  to  the  Cauchy  problem, 

LV  +  A*V  +  VA  +  Q  -  VBR_1B*V  =  0, 


where 


V(T,n)  =  M, 


3V  *  3  1  * 

LV  ■  *  <s  3?V  +  2  tr(PP  i„3n 


T)V, 


and  the  arguments  (t,n)  e  [0,T]  x  R  are  omitted  for  brevity. 
Then,  the  optimal  control  exists,  and  it  is  given  by 


u  °  =  -R  1  Ct )  B  (t,£.)  V(t,£.)x 
t  t  t  t 


Proof:  Proof  is  the  same  as  theorem  3.2.  The  above  Cauchy  prob¬ 
lem  has  a  solution  if  all  of  the  coefficients  of  the  Cauchy  equa¬ 
tion  are  Holder  continuous.  It  can  then  be  shown  that  V  is  non- 
negative  and  uniformly  bounded  on  [0,T]  x  R  [29] .  ■ 

Remark:  The  measurement  data  available  to  the  feedback  controls 
permits  for  a  conditional  estimate  of  zfc  which  is  given  by 

E(z  \V  ,  0  <_  t  <_T) .  The  state  x  also  contains  the  measurement 
t  t  t 

information  about  zfc  as  well  as  yfc. 

Consider  the  problem  of  estimating  the  unobservable  state 
z ,  t  e  [0,T] ,  on  the  basis  of  results  of  the  observation  xfc  and 
y  with  the  following  stochastic  system  of  equations: 

dxfc  *  A  (t,  z^_)  x^d  t  +  B(t)utdt  +  G  (t)dw^  f 


(3-47) 


(3-48) 


dzfc  **  C(t)ztdt  +  D(t)dwt2, 
dyfc  =  F  (t)  z^_dt  +  H(t)dwt3,  (3-49) 

x(0)  =  x  ,  z(o)  =  z  ,  y(o)  =  Y> 

0  0  0 


where  z^  cannot  be  observed  directly.  Let  E(a|Zt»  0  <_  t  <_  T)  be 
A 

E^(ct)  and  a.  Under  the  appropriate  assumptions,  the  conditional 
mean  E(z  |Z  ,  0  <_  t  <_  T)  of  the  given  o-algebra  is  given  by 


A 

dz^ 

a  _  i 

=  C (t) z  dt  +  (E  (z 

A(t,z  ) x  1 

* 

1 

m 

N 

m 

A(t,z  )x 

t 

t  t  t 

t  t| 

•  F(t)zt  J 

t  t  r 

t  t 

.  F^>zt  ■ 

1/G (t) 
0 


1 

dv 

t 

dv2 

t 

(3-50) 


where 


dv1 

t 

l/G (t)  0 

( 

dxt 

- 

(A(t,Zfc) )xfcdt  +  B(t)ufcdt 

2 

=s 

A 

ldV 

0  1/H (t) 

•^t 

F(t)z  dt 

(3-51) 


is  an  innovation  process  with  respect  to  Z^. 

The  conditional  variance  equation  E ( (z  -z  ) (z  -z  ) * | Z  )  becomes 
drt  =  "  2tzt*C(t)*}  dt  +  D(t)  D*  \ '  '  dt 


{  Et[.t 

A(t,zt)xt 

*]  -  zt 

Et  A(t,zt)xt 

F(H)zfc 

F(t)zt 

1/G (t)G* (t) 

0 

* 

<Et 

>  ' 

A(t,Zt)xfc 

0 

l/H (t) H  (t) 

F(t)zt 

L 


37 


Consequently  the  equation  (3-54)  has  been  reformulated  by  ob¬ 
servable  processes  (3-51)  and  (3-52). 
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Comment:  The  conditional  estimate  E [ |  2^ ,  o  <_  t  <_  Tl  is  the 
same  as  because  x^  is  Z  measurable. 

The  innovation  process  in  (3-50)  depends  on  the  control  vari¬ 
able  ufc,  and  thus,  the  separation  principle  could  not  be  applied 
to  verify  the  optimal  control  in  (3-54) .  If  applied  to  the  sto¬ 
chastic  linear  controller,  this  will  show  that  a  lower  cost  can¬ 
not  be  obtained  with  nonlinear  controls.  The  optimal  control 
cannot  be  found  using  any  well-known  methods.  For  practical 
applications  the  most  important  results  desired  are  the  necessary 
conditions  of  optimality  that  can  be  used  for  synthesis  of  opti¬ 
mal  feedback  control  laws.  The  answer  to  this  question  is  yet 
to  be  resolved. 
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4.  SIMULATION  OF  THE  STOCHASTIC  CONTROL  PROCESSES 
WITH  RANDOM  COEFFICIENTS 

In  previous  chapters  the  problem  of  optimal  control  of 
stochastic  control  processes  with  random  coefficients  has  been 
presented.  To  evaluate  the  performance  of  the  optimal  control 
for  each  stochastic  model,  it  is  necessary  to  synthesize  the  con¬ 
trol  law,  the  state  equation  and  the  solution  of  a  nonlinear  par¬ 
tial  differential  equation.  A  discretization  technique  is  used 
to  calculate  the  state,  the  Wiener  process  and  the  optimal  con¬ 
trol  for  simulation. 

A  pseudo-Wiener  process  is  used  for  the  generation  of  the 
Wiener  process;  the  first  method  is  introduced  by  a  Bernoulli 
time  series  [70]  using  pseudo-uniform  random  numbers  between 
0  and  1;  the  second  uses  pseudo-Gaussian  random  numbers  N  (0,1). 

The  solution  of  the  nonlinear  partial  differential  equation 
to  the  Cauchy  problem  of  equation  (2-18)  uses  a  semi -discrete 
system  of  nonlinear  ordinary  differential  equations.  The  semi¬ 
discrete  system  equations  can  be  solved  by  integration  of  the 
ordinary  differential  equations  [71,72,73], 

A  simple  one-dimensional  stochastic  system  has  been  presented 

by  the  state  and  optimal  control  to  the  value  function  V(t,xt,zt>  = 

* 

xfc  Ai(t,zfc)xt  +  A^t,^).  A  Pract^-ca^  application  of  the  theoretical 
extension  in  Chapters  2  and  3  is  presented  here  by  the  landing 
problem  of  the  longitudinal  motion  of  an  aircraft  in  a  gusty  wind. 

A  landing  aircraft  may  be  described  approximately  by  a  second- 
order  differential  equation  with  random  parameters  [44].  Certain 


i  ****** 
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information  data  are  derived  by  conditional  estimation  using  the 
given  observations.  On  the  basis  of  observations,  the  con¬ 
ditional  estimate  is  applied  to  derive  the  appropriate  sto¬ 
chastic  models.  This  aircraft- landing  model  is  used  to  illustrate 
the  design  procedure  of  optimal  control  under  the  worst  weather 
situation.  The  random  parameter  is  assumed  to  be  the  result  of 
a  gusty  wind.  The  models  of  wind,  based  on  the  Dryden  model  for 
turbulence  and  its  aerodynamic  effects,  are  used  in  conjunction 
with  optimal-control  design. 

If  the  system  is  subject  to  both  parameter  uncertainty  and 
noise  disturbances ,  control  of  the  dynamic  system  is  treated  by 
stochastic  control  theory.  The  problem  of  controlling  a  longitud¬ 
inal  motion  of  an  aircraft  in  wind  gust  is  very  similar  to  the 
above  problem  [34],  The  statistical  properties  of  uncertain 
quantities,  which  are  Lebesque-measurable  functions  whose  values 
may  range  with  proper  boundaries,  are  assumed  to  be  known,  and  the 
stochastic  model  of  longitudinal  motion  of  aircraft  in  wind  gust 
is  approximated  by  the  second-order  stochastic  differential  equa¬ 
tion.  It  is  also  found  that  the  simulation  results  of  the  motion 
of  aircraft  in  wind  gusts  give  a  reasonable  degree  of  approximation. 
Using  the  theoretical  results  of  Chapter  2  and  3,  the  angle  of 
attack,  the  orientation  rate  of  aircraft,  the  active  elevator  con¬ 
trol  angle,  and  the  active  aileron  control  angle  in  a  gusty  wind 
are  determined. 
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4.1  The  Generation  of  Wiener  Process 

Let  ,?')  be  a  probability  space,  Wfc,  t  >_  0,  be  a 

Wiener  process,  and  F"  be  a  nondecreasing  family  of  sub-c-algebra 
of  P'.  Then,  the  Wiener  process  has  the  following  properties: 

i)  E(w  -w  If'  )  =  o,  E((W  -w  )2|f')  =  t-s,  t  >  s,- 
t  s  '  s  t  s  s  — 

ii)  W  -  W  and  W  -  W  are  independent  for  nonover- 
fc2  fcl  fc4  t3 

lapping  intervals  [t2,t^l  and  [ t 4~ t^ ] ; 

iii)  the  trajectories  w  cure  continuous  with  probability 
1; 

iv)  W(0)  =  WQ  =  0. 

The  construction  of  the  Wiener  process  is  most  useful  to 
study  the  stochastic  model.  For  convenience,  the  Wiener  process 
is  approximated  with  the  digital  computer  by  a  pseudo-Wiener  pro¬ 
cess  according  to  the  properties  of  the  Wiener  process.  A  contin¬ 
uous  stochastic  integral  for  computation  can  be  approximated  by 
the  following  discrete  Wiener  process: 
t  n-1 

/  g(t)dw  »  I  g(t.)  (w  -w  ) ,  (4-1) 

0  i=0  l+l  i 

where  g(t)  is  some  suitable  function.  The  discrete  representation 

of  (4-1)  should  be  kept  in  an  infinite  sequence  of  independent 

Gaussian  variables  on  71'. 

Consider  first,  the  construction  by  which  a  Bernoulli  time 
series  can  be  made  to  approximate  the  Wiener  process  as  follows: 
with  fixed  ^  >  0,  a  e  [0,T] ,  and  the  interpolation  of  certain 


-/  *■  -  >  .  i- 
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random  sums, 


W^(A)  =  h(^)Z  z.  ,  i  <  t/i(  0<t  <  T, 


(4-2) 


where  z.  is  the  Bernoulli  variable  which  has  a  mean  0  and  a 

i 

magnitude  1.  W^(A)  converges  to  the  Wiener  process  under  proper 

choice  of  the  scalar  h(A)  [53].  If  {z^}  is  a  sequence  of  indepen¬ 
dent  random  variables  such  that 


P[zi=l]  =  P[zi  =  -1]  =  — , 


(4-3) 


and  the  graph  of  W  (A)  is  a  scalar  random  walk.  The  scalar  h(A) 
in  (4-2)  is  selected  so  that  regardless  of  the  value  a,  the  vari¬ 
ance  of  W^(a)  =  t.  Thus  determine  h(A)  as  follows: 


Var[W  (A)]  =  t  =  h(A)  E  ,  var[z.l, 
t  1  ^t/A  1 

=  [h (A) ] 2 (integer  part  of  (t/A) ] , 


£  [h(A)]2. 


(4-4) 


Therefore  h(A)  =  /&,  and  Laplace's  theorem  states 


n  n  -*■  00 

/n  (  I  z.)  - ¥  N(0,t) 

i-1  1 


(4-5) 


If  n  is  t/A,  then  h(A)  *  /t/n  and  letting  a  o, 


W  (A)  sr  h(A)  Z  z.  *  /t/n  (  I  z,). 

i-1  1  i=l  1 


(4-6) 


Since  z.  are  independent,  for  t.  <  t  <  ...<  t  =  T,  w.  ,  W  -  w  , 

i  1  2  n  *2  ti 

...,  W  -  W  are  statistically  independent  and  the  stochastic 
n  fcn-l 


process  W  (A)  converges,  as  i  +  0,  to  the  Wiener  process  W  . 

U  t 

If  there  exist  independent  uniform  random  variables  r^,r  ...» 

r  between  0  and  1,  define  the  following  random  variables  as 
nxm 


-l: 


0  <  r .  <  0.5 

—  l  — 

0.5  <  1.0,  i  =  1,2,...,  nxm. 


Hence,  p(x^=l)  and  p(x^=-l)  are  1/2.  The  approximate  Wiener 
process  is  given  by 


W 


t . 


nxm 


(4-7) 


t.  V  nxm  (  Zi] '  j  =  l/2,...,m,  i  =  l,2,...,n  x  m, 

3  1=1 


where  n  x  m  =  t./A  is  total  number  of  random  variables  z  between 
1  l 

t.  and  0,  and  a  is  the  small  time  interval  t.  -  t.  ..  Equation 
1  3  D-l 

(4-7)  is  equivalent  to 

n(m-l) 

W  =  /a  (  i  z.)  +  W  .  (4-8) 

=  •  m  X  t  — 

J  1=1  3-1 


Figure  4.1  shows  the  normalization  curve  of  (4-8)  for  t  £  [0,1], 
a  =  0.001,  n  =  1000,  m  =  10.  10,000  total  uniform  pseudo-random 

variables  are  used  to  generate  1,000  discrete  Wiener  processes. 

Let  W.  be  a  normal  random  sequence.  In  this  case,  (W^  -  w  ) 

3  fc3  Vl 

has  the  mean  0  and  variance  (t.  -  t.  ,).  Hence, the  standard 

3  3-1 

Brownian  motion  is  always  calculated  by  simply  taking  the  linear 


function  of  (W  -  W  ).  If  (W  -  W  )  =  W.  has  the  distribu- 
t.  t .  ,  t .  t .  ,  i 

3  3-1  3  3-1  J 

tion  N(0,t.-t.  )),  W'  =  W . / (t . -t .  ,)  has  the  distribution  N(0,1). 

3  j-1  j  j  j  j-1 

Table  4.1  shows  the  distribution  of  WC  for  1000  normal  random  vari- 

3 

ables  in  the  range  of  -3.5  <  WC  <  3.5  with  the  distribution  of  WC 

3  3 


being 


Figure  4.1.  Normalization  curve  of  the  pseudo-Wiener  process 


45 


TABLE  4.1  THE  DISTRIBUTION  OF  THE  NORMALIZED  PSEUDO-WIENER 
PROCESS  FOR  1000  RANDOM  NUMBERS 


Range 

Numbers 

Probability  of  Each  Range 

-3.5 

<  w.' 
-  3 

< 

-3.0 

1 

0.001 

-3.0 

<  W.' 
~  3 

< 

-2.5 

10 

0.010 

-2.5 

<  W.' 
-  3 

< 

-2.0 

0 

0.000 

-2.0 

<  W.' 
-  3 

< 

-1.5 

41 

0.041 

-1.5 

<  W.' 

-  3 

< 

-1.0 

102 

0.102 

-1.0 

<  W.' 
-  3 

< 

-0.5 

204 

0.204 

-0.5 

<  W.' 
“  3 

< 

0 

0 

0.000 

W.' 

3 

= 

0 

251 

0.251 

0.0 

<  W." 
3 

< 

0.5 

0 

0.000 

0.5 

<  W." 
3 

< 

1.0 

217 

0.217 

1.0 

<  W.' 
3 

< 

1.5 

126 

0.126 

1.5 

<  W." 
3 

< 

2.0 

37 

0.037 

2.0 

<  W.' 
3 

< 

2.5 

0 

0.000 

2.5 

<  w" 
3 

< 

3.0 

10 

0.010 

3.0 

<  W.' 
3 

< 

3.5 

1 

0.001 
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P (-3  <  W.'  <  3)  =  0.998  s  1.0. 

3 

As  illustrated  by  Figure  4.1  the  density  and  the  results  of  Table  4.1, 
it  is  a  good  approximation  for  1000  discrete  Wiener  process. 

These  random  variables  are  checked  for  the  student  t-distri- 
bution  which  (n-1)  degree  of  freedom.  The  normalized  wiener 
increments  W_j  in  Table  4.2  is  given  by  the  t-test.  All  t  values 
in  Table  4.2  are  less  than  the  critical  values,  and  therefore, 
the  results  satisfy  the  t-distribution.  Hence,  the  pseudo-Wiener 
process  is  assumed  to  be  equivalent  to  the  Wiener  process. 

TABLE  4.2  THE  RESULTS  OF  THE  STUDENT  t-DISTRIBUTION 


Numbers  of  Normal 
Random  Variables  Used 

Critical  Values 

The  Given  t  Value 

100 

3.389 

1.3493 

200 

3.389 

1.5315 

500 

3.310 

1.7067 

1,000 

3.291 

1.8862 

Apply  the  above  Wiener-process  simulation  results  to  the  sim¬ 
ple  first-order  stochastic  system 


dx  =  A(t)xfcdt  +  G(t)dWt, 


where  the  mean  and  variance  S  of 


x  are 
t 


=  A(t)mfc 

St  -  2A(t)St  +  G (t) 2 


m(  0)  =  m  , 

S(o>  =  S  . 

0 


(4-9) 


The  discretization  of  (4-9)  is 


x  =  (1  +  <5a) x  +  Git.  )  [W  -  W  ], 
i  ti-l  1-1  i  i-l 

i  =  0,  1,  2,  ...,  n  -  1,  5s  t  ■  tt-l'  =  T‘ 

Table  4.3  shows  the  t-test  for  A  =  -0.5  and  -1.0,  G  =  1.0,  xQ  =  1.0, 

S  =0.0  with  the  other  conditions  being  the  same  as  before. 

0 

Table  4.4  presents  the  Kolmogorou-Smirnov  test  results  for  A  = 

-1.0  and  x  =0.5.  The  other  simulation  conditions  are  the  same, 
o 

If  the  Kolmogorcv-Smirnov  statistic  exceeds  the  critical  value, 
then  the  generated  random  variables  should  reject  the  hypothetical 
distribution.  Test  results  are  less  than  the  critical  values  that 
have  the  significance  level  of  0.01  [74).  Formally  it  turns  out 

that  most  of  the  entries  in  Table  4.4  passed  the  Kolmogorov-Smirnov 
test. 

The  different  generation  of  the  Wiener  process  W_^  of  Kolodziej 

[29]  used  pseudo-random  Gaussian  variables  V.  which  are  N( 0,1) 

3 

from  the  IMSL  library  called  GGNML.  Increments  of  the  wiener 
process  Wfc  were  approximated  by  the  formula  dw^f&t)  s  /  ^  t  V  . 

Comment:  Other  generation  methods  of  the  Wiener  process  are  pre¬ 
sented  by  [75,76,77]  using  Walsh  functions  and  Harr  functions. 


i 
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TABLE  4.3  THE  RESULTS  OF  THE  STUDENT  t-DISTRIBUTION  OF  A  STO- 


CHASTIC  SYSTEM  FOR  A  = 

-1.0  AND  -0.5 

The  Value 

Number  of  Normal 

Critical 

of  A 

Random  Variables  Used 

Values  The 

given  t  Value 

100 

3.389 

0.9758 

-1.0 

200 

3.389 

1.3923 

500 

3.310 

1.2077 

1,000 

3.291 

1.7027 

100 

3.389 

0.6712 

-0.5 

200 

3.389 

0.8791 

500 

3.310 

0.4978 

1,000 

3.291 

0.6442 

TABLE  4.4  THE  TEST  RESULTS 

OF  KOLMOGOROV- SMIRNOV 

Sample 

Size  Significance  Level 

Critical  Value  Results 

2 

0.01 

0.929  0.699 

5 

0.01 

0.669  0.00009079 

10 

0.01 

0.486  0.000 

100 


0.01 


0.180 


0.000 
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4.2  Numerical  Solutions  of  Nonlinear 
Partial  Differential  Equations 


The  problem  of  solving  (2-13)  for  given  A^(T/zT),  and 

A2(T,zt)  is  called  the  Cauchy  problem.  The  solution  is  understood 

to  be  continuous  in  Rn  x  [0,T]  and  to  have  continuous  derivatives 
3AX  3A2  3A1  3A2  32A1  32A2  n 

3t  '  3t  '  3^  '  35  '  3535* '  3535*  in  R  x  The  solution 

to  problems  in  (2-13)  may  be  transformed  to  initial-boundary  value 
problems  for  t  replaced  by  T-s,  t,s  £  [0,T] .  Then,  (2-13)  becomes 
the  classical  Cauchy  problem  with  the  initial  conditions  instead 
of  the  terminal  conditions. 

Let 


f  ,  .  .  .  ^2  !V 

3t  k  35  '  35  '  ""  35  ’ 


32A, 


32A„ 


*2v 


3535*'  3535*'  '  3535* 


)  , 


(4-10) 


k  =  1 ,  2 ,  . . . ,  NP , 

denote  the  coupled  systems  of  partial  differential  equations  with 
the  initial  conditions 


Al's=o  kl'  A2  s=0  k2  '  '  ANP*s=0  kNP' 


(4-11) 


where  k^  is  some  constant.  Kolodziej  [29]  proved  that  the  partial 
differential  equation  (4-10)  with  (4-11)  has  a  bounded  unique  solu¬ 
tion. 


The  numerical  solution  of  the  nonlinear  partial  differential 
equation  is  complicated  and  is  a  highly  problem-dependent  process. 
The  semi-discrete  system  of  nonlinear  ordinary  differential 
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equations  is  solved  using  one  of  the  recently  developed  ordinary 

differential  equations  integrator  [71,72].  The  numerical  method 

of  lines  [42]  will  be  used  for  equation  (4-10) .  Describe  the 

finite  difference  approximations  used  by  the  computer  simulation. 

Assume  that  a  user  has  specified  a  time-independent  spatial  mesh 

which  consists  of  a  sequence  of  NS(i3)  points  in  [a,b]  such  that 

a  =  5,  <  £  <  . ..<  5  =  b.  Define  the  mesh  spacinq  as  A.  = 

1  2  NS  l 


?i+i  -  V  for  1  =  1'2'- 


,NS-1 .  Associate  with  this  mesh  the 


functions  A.  . (s) ,  j  =  1,2,...,NP  where  j  is  the  number  of  the 
3  i 1 

partial  differential  equations,  and  i  =  1,2,..., NS.  The  value 
of  the  function  A.  . (t)  at  any  time  t  is  meant  to  approximate 
the  true  solution  value  A  (s,5j)-  To  obtain  an  ordinary  differ¬ 
ential  equation  which  will  determine  A.  . (s) ,  evaluate  the  jth 

1  r  1 

partial  differential  equation  (4-10)  at  5=5^  where  1  <  i  <  NS. 
For  numerical  solution,  evaluation  of  A,  A^,  and 


j.  ih. 

35  (  355  ' 


j  =  1,2,  ...  ,  NP, 


is  necessary.  Let  the  approximations  Aj  ^(s)  denote  A ^ 

For  5=5^  in  the  jth  partial  differential  equation,  approximate 


A .  (s ,  C)  a  A.., 
1  ji 


3Ai(,,>C)  a  Ai(i.l)-Ai(i-1) 

35  a.  +  a. 

1  i-l 

a  3A  (s,5) 

—  ( - 3— - - )  5 

35  35 


(4-12) 


(WV/2  -  (Vi+^i,/  2 


A. 


A ,  .  A . .  -  A  . 


•  i  3 (i+i)  ji  _  ji  j  (i-l ) 


)  ,  j=l,2 . NP. 


i-l 


It  remains  to  consider  the  points  i  =  1  and  i  =  NS.  For  i 


consider  the  approximation  at  £=£ 


A.  (t,£)  s  A  , 

3Aj(t'^  W 

3C  '  ^  ' 

_i  9Aj(t,C)  3  (Aj-3~Aj2)/Al  "  (Aj2~Ail)/Al 
3?  3C  A, 


(4-13) 


For  i=NS,  its  approximation  is  given  by 


A.(t,C)  ■  AjNS- 


9Aj (t,C)  ~  AjNS  ~  A1(NS-1) 

35  ‘  ans 


(4-14) 


9Aj (t,^}  _  (AjNS  ~  Aj(NS-l))  /A  NS  ~  (Aj (NS-1)  ~  Aj (NS-2) )/ANS 

35  3? 

At  this  point,  it  should  be  clear  that  the  finite  difference  approxi¬ 
mations  (4-12),  (4-13),  and  (4-14)  are  substituted  into  (4-10). 

Now,  the  semi-discrete  system  of  NP  x  NS  approximate  equation  has 
the  form 


dA. 


dt^-  =  Fjl(t'Aj-l'  Aj-2 '  Aj-3)f 


dA.  . 

"d?  =  Fji(t'Aj(i-l)'  Aji'  Aj(i+1)K 


(4-15) 


di\ . 

Aj  (NS-l)  '  V5'  i  =  2'3 . (NS-15' 


52 


for  j=l,2, . . . ,NP,  where  A  =  (A^,  A^,  A^p^).  Since  the 

no  longer  depends  on  only  spatial  derivatives,  equation  (4-15) 
is  simply  an  approximating  system  of  ordinary  differential  equa¬ 
tions  that  are  easily  obtained  from  (4-11)  with  A..  =  k. 

j  i 1  s=o  3 

for  i=l,2,...,  NS  and  j=l,2,...,  NP. 

Most  of  the  recently  developed  and  currently  available  ordin¬ 
ary  partial-differential-equation, integration  routines  [71,72,73] 
are  designed  to  solve  the  initial-value  problem  for  ordinary  dif¬ 
ferential  equations  (4-15)  where  A,  3A/3S,  F  are  vector  functions. 


Example  4.1  Let  A(t,£),  £  e  R  satisfy 


ft  +  k  +  2  A<t,$)A  +  1-0  -  A2  =  0, 

3t  2  ^2 


(4-16) 


and 


A  (TfO  =  0  ,  t  e  [0 ,T] 

where  S^is  the  solution  of  the  following  stochastic  process 


Here,  w^_  is  normal  Brownian  motion.  Then,  £  by  itself  is  Gaus- 
2 

sian.  Let  m  and  a  be  the  mean  and  variance  of  £  respectively. 
Choose  a  =  m  -  3c  =  -3o  and  b  =  m  +  3c  =  3c.  Hence,  a  =  -3 

and  b  =  +3.  Therefore, 


Prob. (5  e  [-3,3])  a  0.995  s  1.0. 


The  variable  t  is  transformed  by  T  -  t,  and  then,  (4-16)  becomes 
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=  0.1,  i  =  1,2,...,  61, 

i 

=  0.001,  t  e  [0,1] , 

A  =  A  +  A,  tan  1  (£ )  „ 
o  l 

A  =  3.75, 
o 

Ax  =  1.50. 

Figure  4.2  shows  the  numerical  solution  of  (4-16)  for  the  five 

different  values:  a)  A  =3.75,  and  A,  =  1.50,  b)  A  =3.2,  and 

o  1  o 

A.  =  1.2,  c)  A  =  2.5,  and  A.  =  1.0,  d)  A  =  2.0,  and  A  =  0.75, 

1  o  1  o  1 

e)  A  =1.5,  and  A,  =0.5.  The  solution  A  is  similar  to  the 
o  1 

following  Riccati  equation: 

=  2AqA'-  1.0  +A'*. 

Example  4.2  Consider  the  following  stochastic  differential  equation 

dx^  =  (3.75  +  1.5  tan  ^z^x^t  +  ufcdt,  x(0)  =  1.0,  (4-17) 

dzfc  =  dw^,  (4-18) 

where  is  unobservable  and  wt,  t  e  [0,1]  is  a  wiener  process. 

The  solution  to  the  optimal-control  problem  yields  a  control  ut° 
that  minimizes  the  criterion 

^2  2 

J(u)  =  EC  /  (xfc  +  ufc  )dt  ]. 

0 

According  to  the  results  discussed  in  [29] ,  the  stochastic  control 
problem  in  (4-17)  and  (4-18)  has  the  solution  of  the  form 
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ut°  =  -  V(t,2t)mt, 

where 

*  —  1  ^ 

dmt  =  (3.75  +  1.5  tan  zfc  -  V(t,zt))mt  dt,  mQ  =  x0» 

and  V(t,£),  C  e  R  satisfies 
o 

ri\7  1  rl  \7  1  0 

-g“  +  -  ~2  +  2‘  (3-75  +  1-5  tan  Zt)V  +  l-°  -  V  *  0, 

V(l,£)  =0,  t  e  [0,1] . 

Using  the  simulation  results  in  example  4.2,  Figure  4.3  shows  that 
optimal  control  and  suboptimal  control  obtained  for  (4.17)  with 
(3.75  +  1.5  tan_1zt)  replaced  by  E [ 3 . 75  +  1.5  tan  1zt.]  .  Figure  4.4 
shows  the  sample  paths  according  to  optimal  control  and  suboptimal 
control  in  (4-17). 

4.3  Application  to  an  Aircraft  Landing  Problem 

Landing  aircraft  may  be  described  approximately  by  a  second- 
order  differential  equation  with  random  coefficients  [44] . 

2 

-  k*—  =  r  (o ,  v,  5  ,  6  ,  6  ,  6  ,  5  )u  ,  (4-19) 

dt  1  2  3  4  0  t 

where  h(t)  is  the  altitude;  u^  is  the  altitude  control  signal; 
r  is  a  coefficient  depending  on  air  density  c,  the  flight  velocity 
v,  and  aerodynamic  coefficients  6^,  5^,  5  ,  <5^,  6^.  At  the  begin¬ 
ning  of  the  landing  process,  the  initial  conditions  are  given  by 
h(0)  and  h(0).  The  flight  velocity  v  is  assumed  to  be  constant 
throughout  the  landing  process  with  finite  inverval  of  landing 
time  T.  The  coefficient  r,  which  characterizes  the  objective  of 
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Figure  4.4 


Realization  of  xfc  under  optimal  and  suboptimal  for 
example  4.2 


‘SSSfess1' •- _  >. 


■  *»y*VJt* 
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control,  depends  on  many  factors  that  can  vary  randomly  at  any 
moment  in  time.  If  the  altitude  has  an  additive  noise  process, 
then  (4-19)  may  be  modified  as  follows: 


d2h(t) 


r  (t,zt>ut  +  Nfc, 


(4-20) 


where  n  is  a  white  noise  process.  Letting  the  state  variable 
x^  represent  the  descent  rate  h(t),  the  stochastic  model  in  the 
sense  of  Ito  differential  form  may  be  described  by 

dxfc  =  r(t,zfc)utdt  +  g(t)dwt1,  x(0)  =  xQ ,  x  e  R1 .  (4-21) 

the  equation  (4-21)  is  meaningful  insofar  as  it's  an  integral  equation 
[55] .  It  is  assumed  here  that  the  random  parameter  r  is  dependent 
on  a  certain  unobservable  stochastic  process. 


dzfc  =  C(t,zt)dt  +  D(t)dwt2,  z (0)  =  zQ,  zfc  e  R1, 


and  yfc  is  an  observable  process  satisfying 


dyfc  =  F(t,zfc)dt  +  H(t)dw  . 

12  3 

Here  wt  '  '  wt  are  mutually  independent  wiener  processes. 

The  cost  fuction,  which  depends  on  the  rate  of  descent  of  the 
aircraft  during  the  landing  interval  and  the  control  system  u^,  is 
given  in  the  quadratic  form 


J (u)  =  £[  /  u  2dt  +  Mx  2]  . 

0  t  T 


(4-22) 


The  above  control  system  has  the  structure  discussed  in  Chapter  3 
and  the  corresponding  results  can  be  applied  here.  Assuming  that 


L 
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the  first  two  conditional  moments  of  z  satisfactorily  estimate 
z the  following  approximation  of  (4-20)  can  be  used: 


dx. 


=  r(t,z  ,  F  )u  dt  +  g (t) dw  , 
t  t  c  t 


(4-23) 


where  z.  is  E[z  y  ]  and  r  =  E[(z  -z  )zjy  ]. 
t  t1  t  t  t  t  ;  t 

One  of  the  possible  models  for  the  wind  gust  velocity  is 

suggested  by  Balakrishnan  [11,42]  such  that 


z  «  -  azfc  +  3nt>  a  >  0,  6  >  0, 


(4-24) 


and  n  is  a  white  Gaussian  noise  with  unit  spectral  density. 

Hence  z  is  Gaussian,  asymptotically  stationary,  with  the  spectral 
density  function  given  by 
,2 


P(ui)  = 


2  2  2  ' 
4tt  'jj  +a 


(4-25) 


where  P(u)  is  called  the  spectral  distribution  of  z  .  Equation 
(4-24)  has  its  stochastic  differential  form  as 


dzfc  =  -oiz^dt  +  Sdw^  , 


(4-26) 


and  the  observation  process  is  assumed  to  be  of  the  form 


dyt  =  fz^dt  +  dw^' 


(4-27) 


12  3 

Here  ,  w  ,  w  are  assumed  to  be  mutually  independent  Wiener 
processes.  Now  the  suboptimal  filter  for  z^  reduces  to  the  Kalman 
filter,  and  the  conditional  density  with  respect  to  V  is  Gaussian. 


It  follows  that 
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(t.z  ,r  )  =  /  r(t,0)- 

t  t  ~ 


(3-z  ) 

-  EXP  [ - ]d©  , 

/i^r  2 r 


(4-28) 


where 


dzfc  =  azfcdt  +  I^fdv  , 

drfc  =  (-2arfc  +  s2  -  f2rt2)dt, 

d\>t  =  dyfc  -  fzfcdt. 

The  optimal  control  is  given  by 

Ut°  =  -  V(t,Zt,r  )xfc,  (4-29) 

where  V(t,zfc,r  )  is  a  solution  to  the  Cauchy  equation  of  the 
following  form: 

L ( V) -  V2  r2  =  0,  V(T,£,n)  =  M.  (4-30) 


Here, 


IV  3V 

L(V)  =  —  -  -  +  (-2a -.+ 

t  £  (0,T)  ,  R. 


Figure  4.6  shows  the  numerical  solution  to  the  aircraft  land- 

9  "> 

ing  problem.  Here,  a=  -1.26  ft/s ec~,  6  =  1.12  ft/sec1-,  f  =  1.0, 

g  =  0.0,  r(t,zt)  =  6.37  tan  2(8.5  zfc)  ,  Ar  =  0.5,  a  ^  =  o.l,  ^  =  1.0, 

t  £  (0,100]sec,  zq  =  1.0,  Xq  =  -15.0  ft/sec,  and  M  =  0.0005.  Each 

2 

Wiener  process  of  w  is  generated  from  10  pseudo-random  Gaussian 
variables.  The  optimal-control  signal  and  the  optimum  trajectory 
are  plotted  in  Figures  4.6  and  4.7,  respectively.  For  safe  comfor¬ 
table  landing  the  altitude  h(t)  of  the  aircraft  is  often  described 
by  an  exponential-linear  flare  path.  The  results  shown  in  Figure  4.7 
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is  almost  the  same  as  the  expoential-linear  flare  path  so  popular 
in  landing  systems. 

4.4  The  Control  Problem  of  Longitudinal 
Motion  of  an  Aircraft  in  Wind  Gust 

The  control  of  dynamical  systems  which  contain  uncertain  ele¬ 
ments  and  are  subject  to  uncertain  inputs  may  be  treated  by  sto¬ 
chastic  control  theory.  Leitmann  [32,33]  presented  studies  of  un¬ 
certain  dynamical  systems  which  have  an  unstable  mode,  and  he  treat¬ 
ed  the  estimated  state-feedback  control  to  assure  ultimate  bounded¬ 
ness  of  given  states  for  a  Lyapunov  function.  The  purpose  of  this 
section  is  to  develop  systematic  techniques  for  the  design  of  an 
optimal- control  law  which  will  minimize  the  effect  of  the  wind 
disturbances  and  noises  to  an  aircraft  in  a  gusty  wind.  The  con¬ 
trol  of  dynamical  systems  which  include  unknown  parameters  and  are 
dependent  on  uncertain  quantities  are  estimated  by  an  appropriate 
filtering  algorithm.  Even  with  a  stochastic  model  of  the  wind  gusts, 
there  still  remains  the  nontrivial  work  of  computing  the  statistical 
response  of  an  aircraft  flying  through  such  a  wind-gust  fieic.  In 
virtually  all  analyses  of  aircraft  wind-gust  response,  which  this 
author  has  found,  the  incremental  aerodynamic  force  at  each  point 
on  the  aircraft  are  assumed  to  depend  lineraly  upon  the  wind-gust 
velocity.  While  such  assumption  may  be  somewhat  questionable  in 
general,  its  use  in  the  Dryden  model  for  turbulence  yields  a  reason¬ 
ably  accurate  description  of  the  wind  gust  effects. 

Consider  the  control  problem  of  longitudinal  motion  of  an 
aircraft  in  wind  gust  [34].  Let 
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x^  =  angle  of  attack 
0  =  orientation  of  aircraft 


=  active  elevator  -  control  angle 
=  active  aileron  -  control  angle 
=  Wiener  process 

For  such  system,  a  stochastic  dynamic  equation  of  the  motion  of 
the  aircraft  is  given  by 

dxfc  =  A(t,zt)xtdt  +  B(t,zt)utdt  +  G(t)dw^,  (4-31) 

x(°)  =  xQ,  xte  R2, 

where  A(t,z^_)  and  B(t,z^)  are  uncertain  parameters,  and  models 

the  fact  that  the  angle  of  attack  x^  and  orientation  of  aircraft 

0  are  difficult  to  measure.  Let  the  presence  of  the  stochastic 

process  z  model  wind  gust,  where  z^  satisfies 
t  t 

dz  =  C (t) z  dt  +  D(t)dw  1,  z(0)  =  z  ,  z  e  R1,  (4-32) 

w  u  t  U  u 

with  observation 

dyfc  =  F (t) z^dt  +  H(t)dwfc2,  y(0)  =  0 ,  yfc  t  R1 ,  (4-33) 

where  C(t),  D(t),  F(t),  H(t)  have  appropriate  dimensions  and  wfc, 

1  2 

,  wfc  are  independent  Wiener  processes. 

An  approximate  description  of  (4-31)  may  be  given  by  the 
following  stochastic  differential  equation: 

dx  s  A (t , z  ,  y  )x  dt  +  B (t, z  , Y  ) u  dt  +  G(t)dw  ,  (4-34) 

■ff  ^  ^  | 

where  xt  =  [x^x^,  ut  =  tu1'u2]  ,  and  zfc  is  E j  ]  and  y  is 

*  2 1  ~  - 
E((z  -z  )  \Y  I.  Here,  A(t,z  ,y  ),  and  B(t,z  ,y  )  are  given  by 
t  u  C  t  t  t  t 


r 
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A(t,z  .y  >  =/  °°A  ( t ,  £ )  — - —  EXP  (~g  (£~z  jV'^dS  , 

-  ^rt  2  *  t 

B (t, z  , Y.)  =/  ”e(t,C) - - —  EXP  (~~  (C-z.  )2Y.  _1)dC  .  (4-35) 

t  t  -<*>  rz -  2  t  t 


*^2tty  . 


The  conditional  expectation  z  and  the  conditional  covariance  Yt 
are  given  by  the  following  equations: 

dz  =  C(t)zfcdt  +  y  F(t) (H(t)2)"(dyt-  F(t)ztdt), 

dy  =  (D(t) 2  -  Y  (t) 2  P(t)2  (H (t) 2 ) _1  +  2C(t)Yt)dt,  (4-36) 

*0  =  E(zo}  '  Y0  =  COV(Z0>* 

★ 

If  the  state  xfc  =  (x^x^  is  observable,  the  minimization 
of  the  following  cost  function  J(u)  can  be  made: 


J(u)  «  E[/T  (x.  Qx.  +  u  Ru  }dt  .  (4-37) 

o  t.  z  z  z 


Then  the  optimal  control  ut  satisfies 

ut°  =  -R_1B  (t,zt,Yt)  (A1(t,zt,Yt)xt  +  i  A2(t'zt'Vt) > '  (4_38) 

where  A  (t,z  ,Y  )  and  A  (t,z  ,Y  )  are  solutions  to  the  following 
1  t  t  2  t  t 

nonlinear  partial  differential  equations: 

A,  =  -A  A  -  A, A  +  Q  -  A,BR_1B  A  -  (FC)  rg  A 
1  111  1  9£  1 

-  i  V 

i2  -  -V  -  ''25"'1®**!  -  (F{>^  A1  -  I  (r'2F2  A  V 


-  (D2  -  n2F2(H2)_1)^-A  . 

an  2 

Here,  the  arguments  (t,£,n)  are  omitted  for  brevity,  and 


(4-39) 


.  _ _ • 
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A1(T,5,n)  =  0,  A2(T,£,n) 


0,  E,  e  R  ,  n  e  r 


* 

For  the  case  when  the  state  x^  =  tx1>x2^  is  unobservable, 
the  control  problem  has  been  solved  for  a  linear  system  with  quad¬ 
ratic  criterion  having  random  coefficients  [29] .  Consider  the 
following  observation  equations : 


dyl 

ii 

al  0 

X1 

dt  + 

ei  0  ' 

,  2 
dw^ 

CM 

>1 

,°  a2 

■  X2. 

,°  e2. 

dw  ^ 
t 

(4-40) 


2  3 

where  w^  and  w  ,  t  e  [0,T] ,  are  Wiener  processes  and  are  independent 
of  wfc  and  w^1,  respectively. 

Let  1)  to  3)  in  Chapter  2  be  satisfied  and  also  assume  that 


P[ 

Xx  (o) 

< 

kl 

y1(o) 

X2  (0). 

•  k2. 

y2(o) 

]  is  p-a.s.  Gaussian.  Then  in  and  T  , 

t  t 


t  e  [0 , T] ,  are  unique,  continuous,  S^-measurable  solutions  to 


ctai  = 

A(t,z 

,y)  m 

t 

t 

t  t 

~  * 

dr  = 

(Ar 

+ 

r  a  + 

t 

t 

t 

dvt  = 

dyi 

- 

aimidt 

dy2 

.a2m2dt 

m  = 

0 

E[xo 

1 

Y^O) 

,y2(°) 

- t' 't'  t 

*  *  *  -i 

j  -  T  a  (ee  )  aF 
t  t 


*  *  -l 

Ffca  (ee  )  d  v^, 


(4-41) 


★ 

=  E  t(x  -m  ) (x  -m  ) 

0  0  0  0 


y1(o) 

y2<o) 


] , 


where  v  is  a  wiener  process,  and  a  = 

al  0 

,  e  = 

el  0 

1°  a2 

i°  6  2 

Here,  the  equation  (4-41)  stems  from  the  equations  (4-34)  and 
(4-40) . 

There  is  a  parameter  of  particular  significance  to  aircraft 
and  missiles,  namely  the  angle  of  attack  defined  by  the  following 
relation: 

a  =  tan  1 (^) , 

where  w  and  u  are  velocities  of  X  and  Z  directions,  respectively. 
This  is  the  angle  between  the  relative  wind  and  the  longitudinal 
axis  resolved  into  the  XZ-plane.  A  frequently  accepted  alteration 
facilitating  the  mathematics  is  the  approximation  of  constant 
forward  speed.  Certainly  aircraft  are  capable  of  flight  over  a 
very  wide  speed  range  but  the  dynamic  motions  that  are  most  de¬ 
sirable  for  analysis  occur  in  a  relatively  short  period  of  time  - 
on  the  order  of  a  few  seconds  -  during  which  no  appreciable  speed 
change  takes  place.  In  face  axial  accelerations  of  the  order  of 
which  aircraft  are  usually  subjected  do  not  significantly  affect 
their  dynamic  motion.  Under  this  assumption,  the  angle  of  attack 
is 

—  1  w 

a  =  tan  ( - - — -  ). 

constant 

Aerodynamic  forces  and  moments  are  strictly  speaking  functions  of 
certain  state  variables.  Consider  for  example  variable  angle  of 
attack  a(t),  -t  <  ;1(t)<_  it,  on  wing  with  velocity  w(t)  and  u(t). 
Using  aerodynamic  derivatives  or  stability  derivates  [78]  with  re¬ 
spect  to  the  longitudinal-motion  variables,  expressions  of 
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A(t,zfc)  in  equation  (4-31)  for  the  aerodynamic  forces  and  moments 
may  be  approximated  by  A(t,tan  *z  ) . 

Computations  are  carried  out  for  the  following: 

,  '  -2.03  -  1.015  tan_1z  1.00 

A  = 

-8.57  -  4.285  tan  z  -2.75J  , 

_  _  -0.021  -0.156' 

D  — 

-1.82  -0.550 


G  = 

0.012' 

a  = 

1.0 

0 

e  = 

’  0.01 

0 

0. 051 

r 

,  0 

1.0, 

/ 

0 

0.01, 

C(t)=  -1.256673  ,  D (t)  =  1.1209982, 


F(t)  = 

1.0 

H  (t) 

=  1.0 

z(0)  = 

0.1 

t  £ 

[0,10] sec  , 

Q  = 

1.0  0.0 

R  * 

R2  0.0' 

0.0  1.0 

t 

,0.0  R, 

The  cost  function  is  chosen  for  approximating  the  aircraft's  weight 
and  for  control.  For  the  case  of  longitudinal  motion  of  an  air¬ 
craft  in  wind  gusts,  a  reasonable  choice  for  the  cost  function 
is  choosing  the  weight  factors  as 

1  ^  -1  1  J  -1 
—  =  tt  rad  sec  ,  —  =  w  rad  sec 

R1  R2 


Subject  to  the  constraint  of  aircraft  dynamics,  it  is  assumed 
that 


x^CO)  =  0.085  rad,  x2(0)  =  0.01  rad. 

Figure  4.8  shows  the  solutions  of  nonlinear  partial  differential 


70 


equations  for  the  value  function  V  =  A^x^  +  ^2’  trajectory 

is  given  for  the  three  solutions  and  each  solution  is  subject  to 
uncertain  random  parameter  which  is  approximated  by  the  con¬ 
ditional  estimate  E[z\y^]  and  covariance  E[ (z^  -  z^]  2 | y  ] . 

The  attack  angle  relative  to  unperturbed  air  is  determined 
corresponding  to  optimal  control  laws  and  is  plotted  in  Figure  4.9. 
For  the  attack  angle  x  ,  Figure  4.9(a)  presents  the  observable  case, 
and  (b)  presents  the  unobservable  case,  respectively.  The  state 
of  orientation  rate  of  aircraft  relative  to  inertial  line  is  plotted 
in  Figure  4.10.  The  state  trajectory  is  given  for  two  cases. 

Figure  4.10(a)  shows  the  orientation  range  rate  for  observable  and 
Figure  4.10(b)  unobservable  cases. 

The  results  of  the  optimal-control-law  calculations  are  pre¬ 
sented  in  Figures  4.11  and  4.12.  As  will  be  seen  by  comparing 
Figure  4.11(a)  and  (b) ,  the  active  elevator  angle  of  the  aircraft 

for  the  observable  x  and  unobservable  x  ,  respectively,  are  pre- 

t  t 

sented.  Finally  a  comparison  is  depicted  in  Figure  4.14  to  the 
active  aileron  control  angle.  Figure  4.12(a)  and  (b)  also  show 
the  aileron  control  angle  for  the  observable  x^_  and  unobservable 
xfc,  respectively.  Those  comparisons  are  found  to  be  even  more 
smooth  in  the  conditional-Gaussian  filtering  for  the  unobservable 
state  xfc. 


0.8 


Figure  4.8  Solutions  of  Riccati-like-equation  for  the  control  of 
longitudinal  motion  of  an  aircraft 


01  • 


Figure  4.10  The  optimum  trajectory  of  (a)  observable  ( - — )  and  (b)  unobservable 

(-•••)  orientation  range  rate 


control  of  elevation  control  angle  for  (a)  observable  states 
unobservable  states  (....) 


and  (b)  unobservable  states 
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5.  ON  NONLINEAR  FILTERING  AND  TRACKING  OF  TARGET 

Among  anti-submarine  target-motion  analysis  algorithms  for 
mobile  platforms,  one  convenient  mathematical  tracking  model  con¬ 
sists  of  describing  the  target  dynamics  by  a  set  of  state-variable 
equations  that  are  driven  by  a  Wiener  process.  Dynamic  motion  of 
the  moving  target  is  generally  nonlinear  and  unobservable;  in 
passive  bearings-onJ y-tracking  problems  the  observations  us¬ 
ually  appear  in  the  argument  of  an  arc-tangent  function.  The  ex¬ 
tended  Kalman  filter  is  a  popular  metnod  for  treating  such  non¬ 
linear  estimation  problems  [47,48,51,52,55].  However,  if  non- 
linearities  are  sufficiently  important,  the  estimation  error  can 
be  significantly  reduced  through  use  of  a  hig'ner-order  estimation 
technique . 

It  is  the  purpose  of  this  chapter  to  compare  performance 
between  an  extended  Kalman  filter  and  a  truncated  second-order 
nonlinear  filter  as  applied  to  bearings-only-target  tracking  using 
simulation  studies.  An  approach  to  tracking  a  maneuvering  target 
is  also  considered. 

5.1  Modeling  in  Two-dimensional  Space 

A  two-dimentions 1  version  of  the  bearings-onlv-tracking  prob¬ 
lem  is  considered.  A  target  0  is  moving  in  relation  to  an  ob¬ 
server  in  a  viscous  fluid  as  shown  in  Figure  5.1.  The  control 

U  and  U  are  thrusts  in  the  X  and  Y  direction,  respectively. 

1  2  2 
Assume  that  the  magnitude  of  drag  force  is  a-  [velocity  of  0]^ 

[53,54]  where  a  is  a  drag  coefficient;  the  direction  is  opposite 
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Y 


■* 


X 


Velocity 


Figure  5.1  Geometrical  definition  of  vector  tracking 

to  the  instantaneous  velocity  vector.  Under  the  assumptions  of 

a  A  .  A 

planar  motion  and  constant  mass,  define  x^  =  X,  x^  =  X,  x^  =  Y, 
x^  =  Y,  the  state  equations  of  the  target  motion  may  be  approximated 

fay 


X  (t) 

1 

=  x2 (t) , 

x2(t) 

(5-1) 

x3  (t) 

=  x4(t) , 

x .  (t) 
4 

■  X4(t>V>V«/(«+  “T,' 

with  an  observed  process  y(t)  of  the  form 
-1  x3(t) 

y(t)  = tan  •  (5_2) 

Consider  from  (5-1)  and  (5-2),  the  following  stochastic  dif¬ 
ferential  equation: 

dx (t)  =  f(x(t),t)dt  +  B (t) u(t) dt  +  G(t)dwt,  (5-3) 


and  an  observed  stochastic  process,  y(t)  such  that 
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y  (t)  =  h  (x  (t) ,  t)  +  R ( t )  v 


(5-4) 


where  the  dimensions  of  (5-3)  are  equivalent  to  that  of  (5-1)  ,  and 
1  2 

wt  '  wt  »  v^-'  are  mutually  independent  Wiener  processes. 


Here, 


f(x(t),t)  = 


x2  (t) 

-ax^)/  2  2 

2  4 

x3  (t) 


-aX4(t)/x22(t)+x42(t) 


[B (t)  u(t)]  =  [  0  u  0  u  ]  , 

L1  2 


[G  (t)  dwfc] 


9x(t)  0 


0  e2(t) 


dw 


dwt2J 


and 


h(x(t) ,t)  =  tan 


-1  X3(t) 


x3  (t) 


Given  the  stochastic  equations  of  target  motion  and  measurement 
information  in  (5-3)  and  (5-4) ,  consider  algorithms  for  calculating 
the  minimum-variance  estimate  of  x(t)  as  a  functionof  time  and  the 
accumulated  measurement  data.  In  this  case  the  extended-Kalman 
filter  equation  for  (5-3)  is  given  by 

dx(t)  =  f(x(t),t)dt  +  B(t)u(t)dt  + 

P (t)h*  (x(t) ,t) (RR*)_1(y(t) dt  -  h(x(t) ,t)dt) ,  (5-5) 


i 


79 


P(t)  =  f  (x(t),t)P(t)  +  P(t)f  *(x(t) ,t)  +  G  (t)G  (t) * 

X  X 

-P(t)hx*(x(t),t) (RR*)-1hx(x(t) ,t)P(t) , 

x (0)  =  E[x(0)]  =  xQ/  P (0)  =  Covtx0x0*],  (5-6) 

Sf  ( • ) 

where,  for  any  integrable  random  process  x(t),  fx(-)=  —  and  denote 

E[x(t)|y(s),  0  _f_s  t]  by  x  (t)  or  [x  (t)  ]  and  P  (t)  is  the  error 
covariance  matrix.  The  extended  Kalman  filter  is  a  useful  method 
for  considering  nonlinear  estimation  problems.  If  nonlinearities 
are  important,  the  unobservable  state-estimation  error  can  be  re¬ 
duced  through  use  of  higher-order  estimation  methods.  Simulation 
experiments  with  the  truncated,  second-order,  nonlinear  filter  [55, 
79,80]  clearly  show  that  it  may  improve  the  estimates  compared  to 
the  extended  Kalman  filter. 

In  general,  the  conditional-mean  estimate  E^txft)]  satisfies 
the  stochastic  equation 

dE^ [x (t) ]  =  Etff (x(t) ,t) ]dt  +  B  (t)u (t)dt 
+  {EtCx(t)h*(x(t),t)] 

-  Et[x(t)]Et[h*(x(t) ,t)]  }(RR*)_1dvt,  (5-7) 

where  the  innovations  process  corresponding  to  (5-7)  is 

vfc  =  y (t)  -  Efc[h(x(t) , t) ] .  (5-8) 

The  high-order  estimation  equation  for  (5-7)  can  be  approximately 
obtained  as  follows.  .The  truncated  second-order  nonlinear  filter 
was  developed  by  Bass,  et  al.,  [79],  and  independently  Jazwinski 
[55] .  This  filter  carries  to  second  order,  whereas  third  and 
higher  order  central  moments  are  neglected.  Recently  Henriksen  [80] 
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rederived  the  truncated  second-order  nonlinear  filter.  The  follow¬ 
ing  truncated  second-order  filter  estimate  x(t)  in  (5-5)  and  (5-6) 
satisfies 

dx (t)  =  f (x(t) ,t)dt  +  -j  fxx(x(t) ,t)  [CM(P (t) ) ]dt  +  B (t) u  (t) dt 
+  P(t)hx*(x(t) ,t) (RR*)-1{y (t)dt  -  [h(x(t) ,t)dt  (5-9) 

+  \  hxx(x(t),t) [CM (P (t) ) ] ] dt} 


p(t)  =  f  (x(t),t)P(t)  +  P(t)fx*(x(t) ,t)  +  G  (t) G (t) *  (5-10) 

-  P(t)hx*(x(t)  ,t)  (RR*)_1hx(}c(t)  ,t)P(t)  , 

*  ~> 
where  fx(-)  =  9f(‘)/  3x  ,  whereas  fxx(.,.)  is  the  2x2“  matrix 

given  by 


f  =  3.  (f  )  [3f> 

xx  3x3x*  3x*  x;  L  3x, 


3f 

_ > 

3x. 


And  CM (A)  denotes  [A 


matrix  A  where  A  denotes  the  ith  column, 
c . 

l 


3f  3f 

_ x  _ x  . 

3x  3x  ' 

3  4 

]  for  an  arbitrary 
n 


5.2  On  the  Maneuvering  Target  Problem 

Consider  the  target  free  to  maneuver  with  a  finite  continuous 
range  of  acceleration  from  which  a  pilot  can  choose.  Earlier 
work  on  this  area  included  Jazwinski's  work  on  the  limited-memory 
filtering  which  will  respond  faster  than  the  normal  filter  [55] . 
Thorp  described  a  technique  of  switching  between  two  Kalman  filters, 
one  of  which  had  a  fixed  gain  in  order  to  determine  when  a  maneuver 
occurs  [56].  The  deterministic  maneuver  command  u^,  k  =  l,2,...,n 
modeled  as  a  semi-Markov  process  was  introduced  by  Moose  [48,49] . 
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The  work  of  Sworder,  et  al.,  [57,58]  is  aimed  primarily  at  the 
development  of  an  adaptive  control  system  for  linear  systems 
in  which  parameters  may  jump  abruptly.  Friedland  introduced 
properties  of  the  separate-bias  estimation  technique  including 
the  interpretation  of  the  result  as  the  estimation  of  a  constant 
embedded  in  white  noise  [59] .  This  method  of  separating  the  esti¬ 
mation  of  the  bias  variables  that  are  unknown  constants  from 
the  linear  dynamic  variable  was  presented.  Another  useful  ap¬ 
proach  was  presented  by  Mehra,  et  al.,  [60],  Wilsky,  et  al.,  [61], 
and  Chan,  et  al. ,  [62]  using  evaluation  of  the  properties  of  the 

residual  sequence  [innovations  process) . 

Suppose  this  target  is  maneuvering;  then  one  can  examine  the 
possibility  of  the  target  for  velocity  change  and  course  change 
during  the  certain  small  time  interval  At.  Since  the  maneuvering 
is  unknown,  it  makes  sense,  from  a  knowledge  of  the  performance 
capabilities  and  operating  patterns  of  target,  to  analyze  the 
measurement  information  for  observations  given  assumptions  and  cur¬ 
rent  estimate  of  all  relevant  quantities.  If  the  target  is  maneu¬ 
vering  at  an  average  speed  in  a  certain  direction,  the  acceleration 
forces  u(t)  must  be  equal  to  drag  forces  to  each  direction,  or 


V  (t)  a  Et[x2(t)  ’/x02(t)+x^2(t)ls  a  x2(t^x_2(t)+x  ^(t)  ' 


u  (t)  =  a  E  [x  (t)  y  2  .  2 ,  .  ]  s  a  x  (t)  2  .  2 

T-  t  4  x  (tj+x,  (t)  4  x.  (U+x^  (t) 

2  2  4  2  4 


The  filter  performs  well  if  there  are  no  modeling  errors;  then 


the  computed  error  covariance  P{t)  becomes  small,  and  the  filter 
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relies  on  old  measurements  for  its  estimates  and  is  oblivious  to 
new  measurements.  If  an  abrupt  change  occurs,  the  filter  will 
respond  quite  sluggishly,  yielding  poor  performance.  Consequently, 
the  present  filter  has  to  be  sensitive  to  new  data  so  that  maneuv¬ 
ering  of  target  will  be  reflected  in  the  filter  behavior.  Now 
extend  the  model  to  the  case  when  there  exists  the  maneuver  of 
target,  then 


u(t)  =  u  +  C.u 

1  1  al 


\  + 


where  u  and  u  are  accelerations  in  X  and  Y  directions,  respectiv- 
1  a2 

ely.  The  new  target  motion  equation  for  maneuvering  is  given  by 


dXj (t) 

= 

dx2 (t) 

dx3(t) 

dx4(t) 

x2  (t)+x4  (t)  kit 


x4(t) 


-a  x  (t)  v  2  ...  2  ,  J 

4  x _  (t)+x  (t) 

2  4 


r  0  0  0  0 

1  dt  +  Cx  0  f  UaA  dt  +  G1(t)  0 


0  0  l  u 

,0  J 


.0  G2(t)J 


(5-11) 


where  and  £,^  are  certain  random  variables.  The  equation  (5-11) 
is  a  bilinear  stochastic  system.  Here 


,(t)  "  <  T  ,  C  (t)  =  <  T 

1  *■  1  t  >  T  1  '■1  t '  >  T  ’ 
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and  T  and  T  are  random  variables.  Suppose  that  T,T  are  exponentially 
distributed  with  lifetimes  \  and  X',  and  independent  of  xg'wt'vc- 
Process  transitions  are  jumps  so  that  between  jumps  it  remains  in 
specific  states  at  random  times  T,T'.  For  the  linear  dynamic  case 
Davis  [63]  derives  the  optimal,  infinite-dimensional  equations 
for  the  computation  of  the  conditional  mean  of  x(t)  and  the  con¬ 
ditional  probability 


* 

y(s) , 

0  £  s  <  t] 

= 

Pr  [  t  >_  T  |  y  (s)  ,0  <_  s 

<  t'l 

y  (s') , 

0  s^<  t] 

.  Pr[t'>_  T  |  y  (s)  ,0  .5.  s 

I  t'] 

where  T  and  T  are  random  variables,  respectively.  In  this  case,  jump 
processes  have  the  difficulties  ior  the  problem  of  detection  of 
jumps. 

Consider  the  dynamic  process  (5-11)  where  u  and  u  are  ran- 

al  a2 

dom  processes  but  comprise  an  unkno. n  bias  vector  and  depend  upon 
the  following  stochastic  equation: 


where  w 
Here,  G3 
biases . 


du 

= 

ai 

.  dU 

G3(t) 


G4(t) 


dw 


l  dw. 


(5-12) 


3  4 

,  w^  and  mutually  independent  Wiener  processes. 

(t)  and  G„(t)  are  zero,  and  u  and  u  are  constant  unknown 
4  31  a2 
Under  the  consideration  of  (5-12) ,  the  maneuvering  target 


dynamic  equation  is  approximated  by 


(t) 


dx1(t) 

_ 

x2(t) 

dx2 (t) 

-a  x2(t)/x  2(t)+x  2(t)+u  (t) 

2  4  a 

dx^ (t) 

x4(t) 

dx4 (t) 

-o  x4(t)/x  2(t)+x  2(t)+u  (t) 

2  4  a 

du  (t) 
al 

0  1 

du  (t), 

0 

0 

1 

0 
c 
0 
L  o 


0 

0 

0 

1 

0 

0 


dt 


G1(t)  0 


0 

0 


0  G  (t)  0 


0  0  G  (t)  0 


000  G4(t) 


dw 


dw 


dw 


dw 


And  observation  is  given  by 


y (t)  =  h(x(t) ,t)  +  Rv 


12  3  4 

Here,  also  assumed  wt  •  wt  *■  wt  •  wt  »  vt  are  mutually 


Let  (5-13)  denote 


(5-13 


(5-14 

independent 
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dz(t)  =  f(z(t),t)dt  +  B  (t)  u  (t)dt  +  G(t)dw  (5-15) 

Application  of  the  extended -Kalman-filtering  theory  given  ob¬ 
servation  (5-14)  to  the  process  governed  by  (5-15)  results  in  the 
following  equations : 


dz 

t 


dx(t) 

daa(t)- 


f'(z(t),t)dt  +  B(t)u  dt 

T 


(5-16) 


+  y(t)hx*(x(t) ,t) (RR*)_1(y  (t)dt-h  (x(t) ,t)dt) , 

f(t)=  f'  (z (t) , t) Y  (t)  +  Y(t)f'*(z(t) ,t)  +  G(t)G(t)*  (5-17) 
z  z 

-  Y (t) h*  (x(t) ,t) (RR*)_1h  (x(t) ,t)Y(t) . 

X  X 


The  covariance  matrix  Y(t)  in  (5-17)  is  partitioned  as  follows; 


y  (t)  = 


p  !  p 

X  .  XU 
i  a 

- 1 - 

*  ! 

P  P 

XU  | 

ai  a 


(5-18) 


P^  =  autocovariance  of  estimate  of  state  x(t) 

P  =  autocovariance  of  acceleration  estimate  of  acceleration 
a  u  (t) 

-i. 

P  =  crosscovariance  of  x(t)  and  u  (t) 
xu  a 

a 

In  terms  of  the  submatrix  of  (5-18) ,  the  variance  equation  (5-17) 
consists  of  the  following  three  forms 

(5-19) 

yt)  -  fx(x(t),t)Px(t)  +  Px(t)fx*<x(t),t)  +  b(t)Pxu  +  Pxu  b(t) 

a  a 

-Px(t)hx(x(t) ,t) (RR*)"lhx(x(t) ,t)  Px(t)  +  G(t)G(t)*, 
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(t)  =  g (t) g* (t) 
a 


P  *  (t)h*  (x(t) ,t) (RR*)_1h  (x(t),t)P  (t), 

Xu  X  X  XU 

a  a 

(5-20) 


P  (t)  =  f  (x(t),t)P  (t)  +  b (t) P  (t) 

XU  X  XU  u 

a  a  a 

-P  (t)h*(x(t) ,t) (RR*)_1h  (x(t) ,t)P  (t) ,  (5-21) 

A  X  X  AU 

a 

where 


★ 

o 

o 

rH 

O 

G  (t)  0 

b  (t)  = 

t 

g(t)  = 

.0  0  0  1 

■  0  G4(t). 

If  there  is  no  bias,  i.e.,  no  maneuvering,  the  variance  equation 

(5-19)  is  the  same  as  (5-6)  ,  and  P  (t)  =  0  and  P  (t)  =  0. 

xu  u 

a  a 

Mendel,  et  al.,  [81]  show  that  the  estimation  of  the  bias  vector 
u  (t)  can  be  interpreted  as  being  equivalent  to  the  estimation  of 
a  constant  that  is  observed  through  white  noise.  However,  for 
the  dynamic  model  equation  in  (5-13)  observations  do  not  have  any 
terms  of  unknown  maneuvering  bias  vector  u^  except  the  certain  func¬ 
tion  of  state  x(t).  Hence  the  suboptimal  estimate  can  be  found 
by  solving  (5-16)  and  (5-17) .  If  there  exists  maneuvering  of  the 
target, (5-19) ,  (5-20),  and  (5-21)  are  all  coupled  and  hence  have 
to  be  solved  together. 

5.3  Simulation  Experiment  of  Moving  Target 

Initially  the  target  starts  at  X=10,000  feet  and  Y=10,000 
feet.  Assume  that  the  target  maintains  an  average  speed  oi  20*' 2  ft/sec 
on  a  course  45-degrees  and  all  initial  values  of  estimates  replace 
20  different  normal  random  numbers  given  1  percent  of  the  -bove 


specified  values.  All  the  computer  runs  are  made  with  20  different 


runs  to  obtain  the  root-mean-square-error  statistics  associated 
with  each  geometry.  The  results  present  for  root-mean- square 
velocity  and  position  to  the  extended  Kalman  filter  and  the  trun¬ 
cated  second-order  filter,  respectively.  The  results  of  the  esti¬ 
mation  processes  are  presented  by  Figure  5.2,  5.3,  5.4.  In  the 
weighting  of  the  observation  data  it  is  assumed  that  the  noise 
level  on  the  bearing  information  is  0.05  radians  on  the  bearing 
observations.  The  dynamical  noise  levels  are  1  percent  of  velocity. 
The  simulation  results  presented  here  demonstrate  that  the  extended 
Kalman  filter  and  truncated  second-order  filter  are  stable  esti¬ 
mation  algorithms  forbearing  only  target  motion  analysis.  Further¬ 
more,  Figure  5.4  reveals  that  the  truncated  second-order  filter 
is  much  better  than  the  extended  Kalman  filter  for  the  time  duration 
of  the  first  15  seconds.  After  15  seconds  the  extended  Kalman 
filter  performs  better  than  the  truncated  second-order  filter  be¬ 
cause  the  higher  nonlinearity  in  {5.3)  disappears.  It  means  that  the 
target  motion  has  reached  steady  state  after  about  15  seconds. 

The  second  simulation  example  is  similar  to  the  first,  the 
target  maintains  an  average  speed  of  20^2  ft/sec.  Here  if  it  is 
assumed  that  all  system  parameters  are  given  constants,  the  starting 
time  for  turns  and  general  maneuvering  can  be  detected  from  the  inno¬ 
vations  process.  All  conditions  are  the  same  as  the  first,  and 
assume  Xg(0)  =  0,  Xg(0)  =  0  ft/sec2  in  equation  (5-13). 

According  to  equation  (5-13)  and  (5-14) ,  all  cases  are  tested 
with  the  extended  Kalman  filter  and  the  truncated  second-order 


i  * 

i 
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filter.  The  results  of  a  fast  90-degree  right  turn  and  the  maneu¬ 
vering  target  for  the  new  average  speed  30 ft/sec  after  about 
50  seconds  are  presented  in  Figures  5.5,  5.6,  5.7.  In  this  plot, 
the  exact  target  track,  and  the  estimated  track  using  the  extended 
Kalman  filter  and  the  truncated  second-order  filter,  respectively 
are  shown  as  the  estimace  X,  Y  ranges  of  each  direction,  and  the 
estimated  range  of  the  target.  The  presence  of  target-estimation 
delay  to  compare  with  the  exact  target  track  is  visible.  The  simu¬ 
lation  results  show  that  with  abrupt  changes  in  bearing  and  velocity 
of  a  target,  the  results  of  the  extended  Kalman  filter  according 
to  equation  (5-13)  and  (5-14)  indicate  the  effectiveness  of  the 
new  technique  which  promises  improved  tracking  for  maneuvering  tar¬ 
get. 
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6.  CONCLUSIONS 

The  contribution  of  this  dissertation  is  in  the  presentation 
of  the  optimal  and  the  suboptimal  control  for  certain  linear  sto¬ 
chastic  systems  with  known  and  unknown  random  coefficients  in  the 
Bel lman-Hamilton- Jacobi  equation  and  related  algorithms. 

TO  analyze  the  optimal  control  for  a  linear  system  with  com¬ 
pletely  observable  random  coefficients,  the  proofs  of  existence 
of  optimal  control  are  based  on  the  stochastic  version  of  Bellman's 
principle  of  optimality  and  Kolodziej's  results  [29].  The  sub- 
optimal  control  for  the  class  of  certain  stochastic  control  sys¬ 
tems  with  unknown  coefficients  belongs  to  the  coupled  bilinear  sto¬ 
chastic  system  is  also  studied  here.  The  main  assumption  made  is 
that  a  finite-dimensional  filter,  which  is  independent  of  the  con¬ 
trol  and  which  gives  a  good  estimate  of  the  unobservable  coefficients, 
can  be  found.  Structure  of  this  filter  is  used  to  construct  the 
suboptimal  control  in  equation  (3-1).  For  the  quadratic  cost 
function,  the  explicit  formulae  describing  the  control  law  is  de¬ 
rived.  The  gain  of  the  linear-in-state  feedback  is  a  solution  to 
certain  nonlinear  partial  differential  equations. 

It  is  expected  that  the  class  of  stochastic  models,  which 
satisfy  the  assumptions  made  here,  includes  many  of  these  models 
for  which  there  is  no  existing  technique  leading  to  strictly  opti¬ 
mal  results.  The  approximate  stochastic  model  investigated  with 
the  stated  assumptions  produces  a  close  optimal  result.  The 
measurement  information  about  uncertain  quantities  use  Etxfc | 

0  <  t  <  T] ,  but  this  conditional  estimate  could  not  be  used  for 
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the  exact  stochastic  model  in  equation  (3-1)  since  its  innovation 
process  depends  upon  the  feedback  control  law. 

The  numerical  examples  are  provided  to  illustrate  the  previous 
results.  The  generation  of  the  Wiener  process  is  used  for  the 
random-walk  theory  with  the  pseudo-uniform  random  numbers  and  the 
pseudo-Gaussian  random  number  W(0, 1) .  It  turns  out  that  the  pseudo- 
Wiener  process  satisfies  the  general  properties  of  the  Wiener  pro¬ 
cess  and  passes  some  proper  random  testing  methods. 

Numerical  solution  of  the  Riccati-like  equation  to  the  Cauchy 
problem  suggests  the  need  for  more  precise  numerical  methods  to 
be  simulated  by  digital  computer.  In  general,  simulation  results 
of  this  type  of  nonlinear  partial  differential  equation  use  ex¬ 
cessive  computer  time. 

For  bad  weather  conditions  of  an  aircraft  landing  system  in 
wind  gusts,  the  theoretical  results  of  Chapter  3  illustrate  the 
design  procedure  of  the  optimal  control.  It  is  found  that  the  sub- 
optimal  control-design  technique  to  the  stochastic-parameter  linear 
system  provides  an  effective  method  for  choosing  the  appropriate 
stochastic  models.  Moreover,  the  simulation  results  of  altitude 
of  aircraft  is  almost  the  same  as  the  exponential,  linear-flare 
path  for  safe  comfortable  landing.  The  above  method  can  also  be 
applied  to  the  control  problems  of  longitudinal  motion  of  an  air¬ 
craft  in  gusty  wind  containing  a  dynamical  feedback  controller.  It 
makes  it  possible  to  ensure  the  proper  trajectories  of  the  angle 
of  attack  and  the  orientation  rate  for  optimal  control  law  of  the 
elevator  control  angle  and  the  aileron  control  angle  given  a 


reasonable  cost  function.  The  method  proposed  here  cam  also  be 
applied  to  unobservable  stochastic  systems  if  there  exist  the 
conditional  Gaussian  filter  and  simultaneous  solution  of  certain 
nonlinear  partial  differential  equations  corresponding  to  given 
stochastic  models. 

To  approximate  stochastic  models,  the  suboptimal  control  in 
equation  (3-1)  is  a  fruitful  area  for  developing  improved  control 
policies.  The  suboptimal  control  has  considerable  complexity  be¬ 
cause  of  a  linear  function  of  observable  states  and  a  nonlinear 
function  of  uncertain  random  parameters .  However,  this  author 
believes  that  suboptimal  feedback  control  presents  the  most  useful 
optimization  result  in  stochastic  control  theory. 

It  describes  the  nonlinear  target  model  for  an  optimal  de¬ 
cision  algorithm  to  decide  between  maneuvering  and  nonmaneuvering 
targets.  The  extended  Kalman  filter  and  the  truncated  second- 
order  filter  for  implementation  in  anti-submarine  warfare,  target- 
motion  analysis  are  developed  and  presented  in  Chapter  5.  The 
target-motion  equation  for  maneuvering  given  by  (5-12)  is  modeled 
in  the  form  of  a  bilinear  stochastic  system  when  the  unknown  ac¬ 
celerations  are  generated  from  the  equation  (5-12) .  The  optimal 
estimate  cam  produce  the  solution  of  (5-16)  and  (5-17)  which  have 
to  be  solved  together.  It  may  be  possible  to  estimate  adaptively 
the  covariance  Y (t) ,  which  is  dependent  on  the  original  states 
and  a  target  maneuver.  Preliminary  work  using  realistic  computer 
simulation  for  no  maneuvering  indicates  that  the  extended  Kalman 
filter  and  the  truncated  second-order  filter  accuracies  agree  to 
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within  about  3  percent.  Results  shown  in  Figure  5.4  show  that  the 
truncated  second-order  filter  is  much  better  than  the  extended  Kal¬ 
man  filter  for  the  time  duration  of  the  first  15  seconds.  After 
15  seconds  the  extended  Kalman  filter  performs  better  than  the 
truncated  second-order  filter  because  the  nonlinearities  (5.3)  de¬ 
crease  in  significance. 

This  section  discusses  some  special  problems  that  have  yet  to 
be  proven  in  the  area  of  optimal  control.  For  practical  appli¬ 
cations  of  known  results  in  the  area  of  optimal  control,  it  is 
necessary  to  develop  efficient  numerical  technique  for  solving 
nonlinear  partial  differential  equations  in  (2-18)  and  (3-19) 
which  have  multi-dimensions. 

It  would  be  also  interesting  to  study  the  problem  of  optimal 
control  for  stochastic  dynamic  systems  with  discrete  random  coef¬ 
ficients  like  Markov  jump  processes.  For  example  if  the  target 
maneuvers,  a  detector  has  to  estimate  the  unknown  input  accelerations 
of  target  motion.  The  maneuvering  characteristic  of  the  target 
may  be  described  by  the  stochastic  model  provided.  Also  the 
finite-state,  continuous-time ,  Markov  process  with  weak  interaction 
may  be  modeled  as  a  singularly  perturbed  system,  such  as  by  queueing 
network  models  of  computer  systems  which  accentuates  the  need  for 
reduced-order  approximations  of  large-scale  Markov  chains.  The 
theoretical  results  which  are  provided  here  may  be  applied  to 
the  singularly  perturbed  form  to  simplify  the  treatment  of  cost 
equations  and  decentralized  algorithms  in  optimization  problems. 

The  most  important  results  desired  are  with  respect  to  more 
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adequate  mathematical  models  that  can  be  used  for  synthesis  of 

optimal  control  laws.  The  estimate  E [zfc  | V  ]  is  suboptimal  in 

a  sense  that  the  state  variable  x.  also  includes  information  about 

t 

uncertain  quantities.  However,  the  contruction  of  the  conditional 
estimate  is  not  available  with  respect  to  the  o-algebra  because 
of  dependence  on  the  feedback  control  law. 

Also,  it  would  be  useful  to  develop  conditions  of  stability 
for  bilinear  stochastic  systems  governed  by  nonlinear  Ito  dif¬ 
ferential  equations  equipped  with  random  coefficients  which  have 
incomplete  state  information  but  to  make  it  available  for  appropri¬ 
ate  feedback  control. 
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